[OSM-dev] The future of sysadmin

[80n]

1) Remote consoles would seem to be a high priority

2) Each server should have a primary and a secondary sysadmin that can kill
/ restart stuff if the primary sysadmin is not available

3) OSMF should have a board member with executive responsibility for
Operations.


On 7/18/07, Steve Coast <steve at asklater.com> wrote:
>
> All
>
> Despite handing off tile to jburgess and www to TomH, and sysadmin in
> general to NickH I'm still accused of centralising control and being
> evil (I'm looking at you Lars). I don't think I'll ever stop being
> accused of being evil but I will admit to the frailty of not having
> infinite hours in the day.
>
> We need to fix the sysadmin situation.
>
> I'm really sorry if you think I got it wrong, but I did the best I
> could.
>
> When OSM started it ran on my desktop machine at UCL where I was a
> sysadmin. I used to love tuning MySQL and setting up mail servers for
> the little corner of a department I occupied. That was 3 years ago.
> Today, unless I'm much mistaken, my old desktop is still sitting in
> the rack unused. That, or it was sold off.
>
> I somehow managed to convince a very competent sysadmin, Nick Hill,
> to be our sysadmin. He really helped us in setting up the racks we
> got. You may remember that we (Nick Black too) salvaged these from a
> storage depot in the middle of nowhere, arranged hosting (which we
> have till this day) and installed a bunch of machines which NickH
> built. He built them from parts he spent time buying after selling
> off donated old machines we got. He did some really innovative things
> to make this all happen, but I'm not going to tell you about that.
> Suffice to say we owe him a debt of gratitude for all this. Never
> mind that he mapped a significant section of London.
>
> I became aware that I wasn't getting the time to load up planet every
> week to tile and so I handed that off to jburgess and he's done great
> work there. It now works. www similarly with TomH. 80n and ojw have
> requested handing root on dev to spaetz which I'm doing now. That
> just leaves wiki - a VM machine hosted at bytemark for free very
> kindly. So the current set up looks like this for root accesS:
>
> all machines: nick hill, steve coast
>
> additionally:
>
> www: tomh
> tile: jburgess
> dev: spaetz, nick black (for stateofthemap hosting stuff)
> wiki: none
>
> I've shared root with people before on a machine, and it's always
> ended in tears. Someone installs a cpan module, and then someone else
> the same thing as a debian package. Things break. Communications
> fail. So, I've been reluctant to hand off root control simply because
> of the organisational hassles and innevitable failures. It makes it
> simple to hand off machines to those that not only want to sysadmin
> but are deeply involved in the aspects. That is, jburgess does just
> sysadmin tile, hes also fixed lots of cool stuff with the tile
> software itself. Same applies to the others.
>
> This would all be wonderful, but we now have the unfortunate problem
> that Nick Hill has other priorities right now and can't devote as
> much time and energy to OSM.
>
> Even if Nick had infinite time, we're growing at such a rate that we
> need to distribute control of this stuff _anyway_.
>
> While I'd love to give you all root access and break things to your
> hearts content, we have a very serious problem. Only 2 or 3 people
> know where the machines are physically located, how they work and can
> get access to them. Those people are me, nick hill and nick black.
> The people with all the passwords are me and nick hill. So, I've been
> avoiding handing out root passwords like candy not because I really
> hate the idea of other people running the machines - viz handing off
> www an tile - but because if you break anything then I will have to
> drop everything and visit the machines. I just don't have the time,
> closeness or patience for this any more. I've made it very clear to
> the people with root that if they break anything they have to go in
> to central london and reboot the machines. Really.
>
> Of course, this is not sustainable. But, I hope this is all giving an
> insight in to my thinking. Sorry, my evil thinking.
>
> Over in the corner, OSMF has magically raised some pesos for spending
> on, you guessed it, servers. So we're in the joint positions of
> needing a/many sysadmin(s), people who can access the machines,
> people deeply competent, and people to buy and install the new
> machines we need.
>
> Where all the machines are, today, is in central london. I'd love you
> to host machines. Multiple times people have offered to host T at H on
> their servers, and it's not happened. openstreetmap.de sits basically
> idle AFAIK. So - while I respect that you may want to host some part
> of OSM at your machine, and I think it's a good idea, and I'll help
> if I can, and I'm sorry that I'm evil, I have to deal with the
> immediate problem of a lack of sysadmins and people who can fix the
> existing machines, which are spinning right now.
>
> Please help me deal with this problem.
>
> I need people who are very competent with machines in racks, _very_
> competent with linux, who have people skills to communicate with our
> hosts at UCL, who can travel to central London and install / fix
> things that arise and who are trusted by the community. Such people
> might live in Oxford. You will need to help price, buy and install
> new machines and remote power and remote consoles. Really - I cannot
> describe to you how difficult it is going to be to move forward with
> new hardware if we don't have one or two people as described in this
> paragraph.
>
> It will be a pain in the arse - but you'll be helping your 9,000
> friends.
>
> Next, we need more sysadmins. How do we do this? Do we continue with
> a person per machine or function (eg, all the tile machines say, if
> we have more than one?). Do we have some other system? Do we need a
> steering committee for this? How have other projects managed it?
>
> have fun,
>
> SteveC | steve at asklater.com | http://www.asklater.com/steve/
>
>
>
> _______________________________________________
> dev mailing list
> dev at openstreetmap.org
> http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20070719/86a7a2d8/attachment.html>