martini :: blog

martini :: blog http://www.martini.nu/blog/index.html First came the bad mousie, then the one bird. Then the LION, that ate me like a sandwich. en Creative Commons 2013-05-23T15:23:54Z mahlon@martini.nu Get your house in order with Mercurial http://www.martini.nu/blog/2012/07/hg-house-in-order.html At some point in the last few years, I migrated all my code over to <a href="http://mercurial.selenic.com/">Mercurial</a>, and basically never looked back. In addition to code, I've also always maintained separate machine repositories, housing configuration files for miscellaneous colocation servers and home systems -- basically, anything that would consume a bunch of my time to rework from scratch. Make a change to something in /etc? Copy and commit it to the repo, with a comment on what warranted the change. Poor man's configuration management. (Side note: If you have > 1 machines and aren't at least doing this much with config files in ANY <a href="http://en.wikipedia.org/wiki/Revision_control">version control system</a>, why not?) This was obviously pretty far from foolproof, and horribly inefficient. What if I forgot to copy a modified file? If a machine needed rebuilding, how best to deploy the configs in the repo back to the system? The problem grew when I started tracking my home directory. Far more files to manage, and I wanted a "base" set of config files that were slightly different on each of the machines I work on. Branching per workstation seemed the best method to initially deal with this, but I soon found myself spending far too much time in branch merges, rsyncing from repo to homedir and back, and writing helper scripts. There's gotta be a better way! Yeah, there totally is. <h2>Machine configuration</h2> The trick with both machine configuration and homedir management lies in exploiting Mercurials "directory walking" to find a repository. If you're currently in /usr/local/etc and run an hg command, Mercurial checks for a repository in /usr/local/etc, then /usr/local, then /usr, then /. It stops as soon as it finds one. This means that if you initialize a new repository at the root of the filesystem, Mercurial will always find a repository, no matter where you're at. If you're actually working in a "regular" code repository elsewhere in the filesystem, Mercurial will see that first, so it behaves exactly as you'd expect it to. Doing something like an 'hg stat' is of course very slow when walking an entire machine, so there's a second trick -- an .hgignore file that tells Mercurial to ignore everything by default. With this file in place, Mercurial just steps aside -- hg commands only operate on explicitly added files, and it's super quick. A new machine repository is setup like so: <pre><code>$ cd / $ sudo hg init $ echo '.*' | sudo tee .hgignore $ sudo hg add .hgignore $ sudo hg commit -u Mahlon -m "Initial commit of `hostname` repository" </code></pre> And that's it. You'll have to be root to add files to or edit the repo. When there's something you want to track, just hg add it and commit. Even better, a backup of the important config files on a machine just becomes a simple: <pre><code>$ cd /tmp; sudo hg bundle -a `hostname -s`.bndl </code></pre> Or from a remote machine: <pre><code>$ hg clone ssh://root@example.com// example.com </code></pre> Redeploying configs to a fresh system has a few small, but arguably unintuitive additional steps. Mercurial is careful to not stomp on existing files, and cloning to / will give you a 'destination is not empty' error by default. This is a good thing, but we have to work around it by cloning elsewhere, then moving the .hg directory manually to /. There's no need to populate the resulting checkout directory with the files, so we pass the '-U' flag to clone. <pre><code>$ hg clone -U /tmp/example.bndl /tmp/confs $ sudo mv /tmp/confs/.hg / $ rm -rf /tmp/confs </code></pre> The repository state now needs to be resynced before we can use it, late binding the repository to the machine. (Thanks to <a href="http://www.deveiate.org/">Michael Granger</a> for pointing <a href="http://hgtip.com/tips/advanced/2010-04-23-debug-command-tricks/">this</a> out.) Since Mercurial will find the repo in /, you can run these commands from anywhere on the system (that isn't within another Mercurial repo): <pre><code>$ hg debugsetparent tip $ hg debugrebuildstate </code></pre> Finally, extract the files from the repo over the existing system files. <pre><code>$ hg revert --all </code></pre> Fwa-tow! Okay, moving onwards. <h2>Homedir files</h2> With per-machine config repos, you very rarely have a need for branching, or pulling changes from one place to another. It's possible, of course, but I like to think of it as a bonus, instead of a regular way to work with them. My homedir config, on the other hand, is used between no less than 6 different systems. Different operating systems, different versions of applications, and different environments. I really don't want separate repositories for each permutation -- I want to check my homedir out and get to work. The idea here is to have a foundation -- base configurations that are identical between all workstations, and then layer on changes needed for each workstation that can be tracked separately, but still carried with the primary repository. (This method works equally well for two workstations or twenty.) Enter <a href="http://mercurial.selenic.com/wiki/MqExtension/">MQ</a>. MQ has the concept of "guards" -- tags you can apply to individual patches, that alter the default patch stack. If you have a patch (or patches) that change the base configurations for a workstation, you can guard it with an arbitrary label, then select/set that label for the workstations that it should be valid for. Machine "A"'s configurations are completely ignored when on machine "B", and visa versa. I keep one primary repo that all machines with my homedir environment sync from (and to.) Initial setup looks like this -- very similar to the machine repo setup, but with the addition of MQ: <pre><code>$ hg init repo/homedir $ hg init --mq repo/homedir $ cd repo/homedir $ echo '.*' > .hgignore $ hg add .hgignore $ hg commit -m "Initial commit of homedir repository" </code></pre> Before continuing, I recommend setting up a shell alias for working with MQ, as mentioned in the <a href="http://tortoisehg.bitbucket.org/hgbook/1.4/managing-change-with-mercurial-queues.html#id2858016">hgbook</a>. For each workstation, perform the same initial trick as above with the machine repos. You only have to do perform this 'bootstrapping' once per system. Note the use of qclone instead of clone, so we snag the MQ patch repo too. <pre><code>$ cd ~ $ hg qclone -U repo/homedir /tmp/homedir $ mv /tmp/homedir/.hg . $ rm -rf /tmp/homedir $ hg debugsetparent tip $ hg debugrebuildstate </code></pre> Add your dotfiles and whatever else you want as part of your base homedir foundation. <pre><code>$ hg add .i3/* $ hg add .bashrc .vimrc ... $ ... $ hg commit -m 'Added some files for all environments' $ hg push </code></pre> Ok. Now's where it gets interesting. Lets say you're on a machine called "hotsoup", and you want specific configurations that ONLY apply to it. Create a new MQ patch file and guard! You can call it whatever you want, but I like to name both the patches and the guards after the hostname they are supposed to apply to. <pre><code>$ hg qnew hotsoup -m 'Configs specific to hotsoup' $ hg qpop $ hg qguard hotsoup +hotsoup $ mq commit -m 'Initial commit of hotsoup patch' $ mq push </code></pre> Now that the patch is guarded, it won't apply to any machine that doesn't have a matching 'hotsoup' guard. So on the real 'hotsoup' machine, select the guard. You should see the following: <pre><code>$ hg qselect hotsoup number of guarded, applied patches has changed from 1 to 0 </code></pre> You only need to do this once, after the initial homedir repo qcloning. Repeat the process on any number of machines. One guard per workstation, and each workstation will only "see" its specific patch. If you want to commit new home directory files for all workstations, just hg qpop the machine patch and commit as normal. Changes while the patch is applied (hg qpush) will be applied only to that particular workstation. Pretty. Dang. Nice. <h2>Gotchas?</h2> Mercurial doesn't track permissions, other than the executable bit. If some of your tracked files require specific perms, you'll need to save them on checkin and reapply them on checkout. There are lots of tools to do this, depending on your operating system. My preference is <a href="http://people.freebsd.org/~kientzle/libarchive/man/mtree.5.txt">mtree</a> for FreeBSD. You can go a step further and add hooks to the checked out repository's .hgrc file, so retaining and reapplying permissions is automatic. This StackOverflow <a href="http://stackoverflow.com/questions/1289816/can-mercurial-be-made-to-preserve-file-permissions">question</a> outlines that method nicely! Serve an anonymous shell via Tor http://www.martini.nu/blog/2010/06/tor-vbox.html I've been an avid user of <a href="http://www.torproject.org/">Tor</a> since about 2005. Retaining the right of anonymity and individual privacy grows increasingly important as different communication technologies emerge and become ubiquitous, as the recent Facebook <a href="http://www.eff.org/deeplinks/2010/04/facebook-timeline">heat</a> has reminded us. If you don't think it matters, you need to stop what you're doing right now and read Phil Zimmerman's excellent PGP <a href="http://www.philzimmermann.com/EN/essays/WhyIWrotePGP.html">justification</a>. (I am particularity fond of the postcard metaphor.) Done? Okay. Lets move on! While most people use Tor simply for anonymous web browsing, Tor also provides a slick way to host a service (web site, IRC chat server, etc) called a <a href="http://www.torproject.org/hidden-services.html.en">Hidden Service</a>. These services are "hidden" because they are only accessible via the Tor network, and are under the same anonymity umbrella as Tor clients -- unless they accidently expose information about themselves, it is essentially impossible to determine the source location of the service. As it turns out, not exposing information about yourself is a lot more difficult than you might imagine; even for a single process. Hosting a simple web page involves configuring your server signature and potentially customizing error messages. Are you sure any images on that web page don't have embedded EXIF data? What about documents that support author metadata (PDF, DOC, many many many more?) META tags if you used an HTML editor to construct your pages? This got me thinking about how to offer other services to anonymous users. Wouldn't an encrypted backup service be neat? That wouldn't really be practical for a hobbyist (storage needs, Tor network speeds, etc) -- but a generic anonymous shell would certainly be a nice communications tool. You could use it for file storage, chat, message passing, or even as a digital <a href="http://en.wikipedia.org/wiki/Dead_drop">dead drop</a>. If I (or anyone) wanted to offer that as a hidden service, how best to do so while 1) remaining anonymous as a provider and 2) providing default settings that retain Tor ideals (trusted privacy, resistance to traffic analysis, and accessibility for blocked users) to the shell accounts? We're talking about exposing an entire OS, after all. Here's what I came up with. Steel thyself for diagram madness!! <a href="/images/blog/2010/06/tor-vbox.png" class="highslide" onclick="return hs.expand(this)"> <img src="/images/blog/2010/06/thumb_tor-vbox.png" alt="Diagram-o-rama!" /> </a> There are a lot of advantages to this setup. Entire environment snapshot/rollback, complete machine mobility, FreeBSD 'secure levels' for added security, and network obfuscation via <a href="VirtualBox">VirtualBox</a> <a href="http://en.wikipedia.org/wiki/Network_address_translation">NAT</a> and OpenBSD's <a href="http://openbsd.org/faq/pf/">PF</a>. And the biggest disadvantage? Speed. Tor is designed for privacy, not for performance. It's really, really slow. I'm talking 2400 baud modem slow, at times. Again, this isn't a deal breaker if you've got your expectations set. You can help the overall speed of the network, too. More on this later. So, here's a step-by-step for the setup I'm using. <h2>Tor Setup</h2> In your torrc file, you can enable the hidden service via two lines of config. <pre><code>HiddenServiceDir /path/to/service_dir/ HiddenServicePort 22 127.0.0.1:50022 </code></pre> This automatically creates two files wherever your service_dir points to; the unique *.onion hostname that your Tor client is now advertising to relays, and the private key for it. Don't lose this key if you care about hanging on to your *.onion hostname. The HiddenServicePort directive tells Tor that incoming connections to port 22 (ssh) that are coming in via Tor should be redirected to localhost, port 50022. We'll now tell VirtualBox to listen on that port, and do its own forwarding. <h2>VirtualBox Host</h2> One of the biggest reasons to use VirtualBox for this server is mobility. As long as you carry the generated Tor hidden service key alongside your VDI file, you can essentially move this server to any VirtualBox supported host OS, anywhere in the world, and it'll still be accessible via Tor at the same .onion hostname. It reminds me of the inter-dimensional Black Fortress from the old movie <a href="http://en.wikipedia.org/wiki/Krull_(film)">Krull</a>, or the <a href="http://en.wikipedia.org/wiki/The_Dark_Tower_(series)">Dark Tower</a> from Stephen King's Gunslinger series. Some "physical" construct that ethereally shifts locations. Neat-o. Let's call it "Tower" for kicks. VirtualBox also gives us the ability to easily hide behind NAT. The guest operating system receives network information from the host environment, providing an immediate sandbox, and IANA private IP ranges. (More on those specifics <a href="http://www.virtualbox.org/manual/ch06.html#network_nat">here</a>.) First off, lets forward traffic from port 50022 to the virtual machine at port 22, so the Tor network can eventually reach the ssh daemon. <pre><code>% VBoxManage modifyvm "Tower" --natpf1 "ssh,tcp,,50022,,22" </code></pre> Unfortunately, VirtualBox NAT exposes resolvers and search domains via DHCP from the host environment by default, so we require some additional guest configuration. <h2>VirtualBox Guest (aka FreeBSD Host)</h2> The VirtualBox guest OS runs a recent <a href="http://www.freebsd.org/">FreeBSD</a> image, in itself acting as a host for a jail underneath it. The single largest potential for identifying machine location is the internet itself. The local IP address will be 10.0.2.15 (by default), so that doesn't give any location sensitive info away... but what about a simple traceroute? Any outbound connection will reveal the server's gateway IP, and it's simply not feasible to try and subvert this without <a href="https://trac.torproject.org/projects/tor/wiki/TheOnionRouter/TransparentProxy#BSDpf">forwarding all traffic</a> back across Tor. So the first thing we do is block all outbound traffic via PF. I renamed the interface to net0 so we can reference it everywhere, and if we decide to change the underlying virtual device, we only need do so in one location. Here's an example /etc/pf.conf. We're not running ssh on this host, only on the jail inside of it -- so ssh traffic that hits the host is redirected inside to 127.0.0.2. We blindly accept all other traffic -- this is safe as no traffic can reach this machine that is not explicitly port forwarded from VirtualBox. <pre><code>net_if="net0" set optimization normal set block-policy return set loginterface $net_if set skip on lo0 scrub in on $net_if all fragment reassemble rdr pass on $net_if proto tcp to port 22 -> 127.0.0.2 block out on $net_if pass in on $net_if </code></pre> We should also override the dhclient options file at /etc/dhclient.conf, so host information remains private. <pre><code>supersede domain-name "tower"; supersede domain-name-servers 8.8.8.8; </code></pre> Also -- resist the urge to set the timezone! UTC time doesn't give away what part of the world you're in. Here's the /etc/rc.conf that loads up PF, and starts up the jail. (Check the jail(8) <a href="http://www.freebsd.org/cgi/man.cgi?query=jail">man page</a> if you are looking for information on how to create a new one -- that's outside the scope of this already very long post.) <pre><code>hostname="tower-host" ifconfig_le0_name="net0" ifconfig_net0="DHCP" pf_enable="YES" pf_rules="/etc/pf.conf" pflog_enable="YES" sendmail_enable="NONE" tmpmfs="YES" tmpsize="128m" fsck_y_enable="YES" kern_securelevel_enable="YES" kern_securelevel="3" jail_enable="YES" jail_list="tower" jail_set_hostname_allow="NO" jail_socket_unixiproute_only="YES" jail_sysvipc_allow="YES" jail_tower_rootdir="/jails/tower" jail_tower_hostname="tower" jail_tower_interface="net0" jail_tower_ip="127.0.0.2" jail_tower_exec_start="/bin/sh /etc/rc" jail_tower_exec_stop="/bin/sh /etc/rc.shutdown" jail_tower_devfs_enable="YES" jail_tower_fdescfs_enable="NO" </code></pre> What else? We should ensure users can't see what other users are doing via sysctl variables. (These behaviors are carried to the jail.) Also, I set a cron that removes login histories every minute. <pre><code>% cat /etc/sysctl.conf security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_read_msgbuf=0 % echo '* * * * * root rm /var/log/wtmp \ /jails/tower/var/log/wtmp' >> /etc/crontab </code></pre> After all of this stuff is set up to our content, lets lock it down via chflags. <pre><code>% chflags schg /etc/pf.conf /etc/rc.conf \ /etc/dhclient.conf /etc/sysctl.conf /etc/crontab </code></pre> Doing this, in combination with securelevel, will protect the files from modification. Even if an attacker somehow gains root access in this FreeBSD host, they can't disable the firewall or change these files without rebooting to single user mode -- and there's no network in single user mode. This also means: if you want to make changes, you need to be sitting at the VirtualBox console. <h2>FreeBSD Jail</h2> This is finally the environment that an anonymous Tor user will be exposed to. Why the additional layer here? Couldn't you just have people log in to the FreeBSD host? Well, sure. However, good hardening practices include the assumption that the box can be compromised (if not already so), and configure it as such -- just like we assumed someone having root in the host (and chflag'ed accordingly.) The chances of "leaking" identifying information are greater from the guest OS to VirtualBox than they are from a jail to the host that is properly locked down. In addition, it adds an extra obfuscation layer that an attacker would have to break through. Root access in a jail is fairly useless. They can't make any permanent system modifications, nor read any additional system data that a normal user couldn't. They can't even see the current firewall ruleset. They can read other user's homedir data -- but just like the real operator of this service, an attacker would have no information available to determine identity. Some other things to set up: <ul> <li>Bump up the LoginGraceTime in the sshd config, since Tor is slow.</li> <li>Add automatic shell history removal to /etc/csh.logout and /etc/profile, depending on if the default shell is tcsh or bash, respectively.</li> <li>Add a reasonable hostname to /etc/hosts for 127.0.0.2.</li> <li>Disable adjkerntz in /etc/crontab -- time is set from the host.</li> <li>Maybe precreate a bunch of users? If you're invited to a shell server by a pal, and there is only one other account... that would be an identity leak. ;)</li> <li>Install any additional ports/packages you want to offer from the shell. I like ntalk/ytalk!</li> <li>How about a quick script for creating new accounts? Maybe linked to a guest account for anonymous creations?</li> </ul> Here's the entirety of the system rc.conf for the jail: <pre><code>sshd_enable="YES" inetd_enable="YES" syslogd_enable="NO" cron_enable="NO" update_motd="NO" sendmail_enable="NONE" </code></pre> And here is what you'd see if others are logged in. Note that the remote IP will appear to be the VirtualBox host. <pre><code> 3:51PM up 14:53, 1 user, load averages: 0.03, 0.01, 0.00 USER TTY FROM LOGIN@ IDLE WHAT q44r4 pts/0 10.0.2.2 3:51PM - w achgung pts/2 10.0.2.2 2:21AM - - lenny pts/1 10.0.2.2 10:51AM - - </code></pre> <h2>The Easy Way</h2> Instead of re-doing all of that above, you could also just grab my VirtualBox machine that already includes all of it. Fire it up, login as guest, enjoy. The host password is root/root, if you wanted to change it. Get it <a href="http://dl.dropbox.com/u/7234177/Tower.zip">here</a> -- it's about 325M, and should import right into VirtualBox. <h2>Configuring your SSH client</h2> SSH needs to be told to use Tor for connectivity, and for DNS lookups (so it knows how to find a .onion addresses.) The <a href="http://bent.latency.net/bent/git/goto-san-connect-1.85/src/connect.html">connect</a> proxy forwarder can do this really easily. It's available in macports, FreeBSD ports, and Ubuntu's package management. You can also compile it yourself, it doesn't have any funky dependencies. After it's installed, add this configuration chunk into your ~/.ssh/config file: <pre><code>Host *.onion PubKeyAuthentication no VisualHostKey yes Compression yes ForwardAgent no ForwardX11 no PreferredAuthentications password ProxyCommand /path/to/connect -S 127.0.0.1:9050 %h %p </code></pre> ... and that's it. You'll be able to ssh as normal, and if you're connected to Tor, ssh will know what to do with hosts that end in .onion. Make sure that if you're connecting to a .onion host, that you explicitly pass your username -- otherwise, ssh defaults to using the username of the currently logged in user. <h2>What else?</h2> If you have even a little bit of bandwidth to spare, please consider being a Tor <a href="http://www.torproject.org/docs/tor-doc-relay.html.en">relay</a>. It can <a href="https://trac.torproject.org/projects/tor/wiki/TheOnionRouter/TorFAQ#DoIgetbetteranonymityifIrunarelay">arguably</a> increase your own privacy while using Tor, and it improves the overall speed of the Tor network. It's easy to limit what kind of traffic you'll allow your relay to pass, and you can even bandwidth limit it. Have I forgotten anything? Could privacy be improved in this environment for end users and/or the server operator? Let me know -- I'll update the VirtualBox export! How would you go about doing this? Publishing a REST API? http://www.martini.nu/blog/2010/05/rest-doing-it-wrong.html Aww, <a href="http://www.facebook.com/">Facebook</a>. We all know your backend is a <a href="http://www.makeuseof.com/tag/facebook-work-nuts-bolts-technology-explained/">Frankenstein</a> of legacy foundation overlaid with new-and-cool features that could probably make the <a href="http://www.livejournal.com/">LiveJournal</a> codebase actually appear organized. In day to day usage of your service, it doesn't really matter. Somehow it is all glued together into a cohesive experience for the end user, and I rarely notice the .php extension in the address bar. And your user interface team remains so very good. (I mean that.) It shows, however, when a developer cracks the hood of your <a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a> API. So, so ghetto. And you're not alone. I'm noticing a disturbing trend with API designers. I'm looking at you, <a href="http://www.twitter.com/">Twitter</a>. Don't think you're getting away with it either, <a href="http://www.foursquare.com/">Foursquare</a>. You guys are the cool kids. You shouldn't have a bunch of cruft lying around. You still can fix this, and your developer userbase would actually appreciate you for it. Here's the gist: REST is designed to piggyback on HTTP, and HTTP is surprisingly full featured. You get an amazing array of features available to you as part of the protocol, like document caching via last-modified times, byte-range fetches, pipelining requests, a huge assortment of authentication options, and content negotiation. You can pick and choose what you want to support, and HTTP provides for a standard way to accomplish it. HTTP compliments the REST design philosophy, and if you're creating a REST API that doesn't take advantage of that fact (or worse, re-invents portions of it), then you just look either lazy, or like you never read any HTTP RFCs. <h2>HTTP verbs</h2> A verb is a synonym for an HTTP method. Given a URI, what is your intent? Reading it? Adding new content to it? Removing it altogether? HTTP has a lot of verbs. But for purposes of REST, it's pretty straightforward. We'll just use 4 of them, which should cover 95% of the REST use case: <pre><code>GET Read from a resource POST Create a new resource PUT Update an existing resource DELETE Remove a resource </code></pre> Foursquare gets this largely right (GET a list of checkins vs POST a checkin), as does Twitter. Both strangely omit the PUT verb completely though, using POST for all writes -- regardless of whether the write is creating new content or updating existing stuff. Even Wikipedia confuses what PUT should do, but the HTTP <a href="http://www.ietf.org/rfc/rfc2616.txt">RFC</a> is clearer. PUT is for updates! There is even a handy dandy error code dedicated to HTTP method handling. As an example, if you try and POST/PUT to a resource that should always be read only, your server should respond with a 405 Method not allowed, along with an Allow header that lists the verbs you can use at that URI. See? HTTP takes care of you like a gentle lover, and any decent HTTP client (web, desktop, or whatever) should handle your API in the correct fashion. With the exception of adding video, Facebook's API completely ignores the concept of verbs. Everything is a GET request, and the verb is embedded into the resource. getSomething. addSomethingElse. At that point, you're immediately not using REST. You're just publishing a loose collection of random URIs and calling it an API. You might as well be using <a href="http://en.wikipedia.org/wiki/SOAP">SOAP</a>, because that's what you just reinvented, without all the XML suck. Oh, wait... that brings me to <h2>content negotiation</h2> Everyone is getting this wrong. Here's the litmus test -- if you need to embed your desired response format in the resource URI, the API is being lazy. <pre><code>https://api.facebook.com/method/users.getInfo?uids=4&format=JSON http://api.foursquare.com/v1/user.json http://api.twitter.com/1/statuses/update.json </code></pre> Again, HTTP supplies a standard method to do this. A client can send an Accept header, that can even specify order of preference it can deal with the data. All format types are MIME. Realistically, most clients will have such a strong preference for a specific serialization format, that they won't ever use Accept parameters or qvalue weights. They'll just say "gimme the resource in application/json format!" and the server will say SURE. Alternatively, the client requests a format that the server can't comply with, and lo and behold -- there's an error status for that, too -- 406 Not Acceptable. Servers, if they want to be -really- nice, can provide alternate resource URIs in the body of the error that explain to the client what to do instead, in the originally requested format! If the APIs actually used this handy header, then they could rapidly start supporting things like compressed JSON streams, or new serialization formats on the fly, all while continuing to support the other client applications out there that don't support the new and fancy formats. This opens up the API to a whole audience of clients that might normally not be interested, while keeping your resource URIs completely unchanged and static. Facebook's new Graph API gets around the whole mess of serialization formats by only supporting JSON. Ever. Want something else? Working with a language with subpar or no native JSON support? Too bad. I personally really like JSON, but why would you intentionally cut off other formats when it would be so easy to support? <h2>POST (kinda)</h2> I update a <a href="http://en.wikipedia.org/wiki/Finger_protocol">plan</a> file regularly, and script updates in my plan file to Twitter. (Then a FaceBook application syncs from Twitter.) This way, I can continue using my "update the world before social networking existed" technology, while still telling Mom what's up on FaceBook. That's right. Get off my lawn!! I like calling Twitter updates 'twits'. Twits + plan file? Plits! A CLIENT IS BORN. Here's what a status update URI looks like to Twitter. <pre><code>http://twitter.com/statuses/update.json?status=I%20eat%20burritos&source=plit </code></pre> Ok, what the hell is this lame excuse at REST? Why do I have to include my 'source' in the URI, when there is a perfectly good User-Agent header? Why on earth must I embed the content of the POST (thank you, it IS at least a POST) in the URI after encoding it appropriately? This is HTTP dammit! It has a very solid concept of a content body! Just look at how pretty this would be if REST ideals were actually applied. <pre><code>POST /statuses/update HTTP/1.0 Accept: application/json User-Agent: plit Content-Type: text/plain Content-Length: 13 I eat burritos </code></pre> What if someday, Twitter suddenly supports posting audio updates to your feed? Like a giant social-networking Nextel walkie talkie! (Please, for the love of all that is holy, lets hope this never happens.) Watch the magic. <pre><code>POST /statuses/update HTTP/1.0 Accept: application/json User-Agent: plit Content-Type: audio/x-wav Content-Length: 273 [audio stream here] </code></pre> Notice that the API is essentially unchanged! Twitter could simply reply with what formats it can support, and the client can do the brainy work of converting the audio to a supported format. How would this ever work with putting content in the URI? Much better dev experience, and less work for developers making clients for your services. A clear path for you, the service publisher, to optionally provide all sorts of rich media options, while supporting protocol standards between clients. DARE I SAY A BETTER WORLD. Please guys, lets take advantage of HTTP if you continue using it as a transport. Or if you stay the course, stop calling what you have REST. It's misleading and just plain... not. A modest attempt to de-dumb scp http://www.martini.nu/blog/2010/05/scp-petpeeves.html The title of this post isn't entirely fair; scp is an awesome utility, and is probably one of my top 10 typed commands in a given day. (Sadly true, I shuffle files around a lot). Because of how often I use it, however, I do have some long standing issues that have always striked me as just plain silly behavior. <h2>flag consistency</h2> scp strives to be consistent with its insecure predecessor, rcp, just like ssh shares many of the same flags as rsh. rcp has been around since 1982, and was largely obsoleted when ssh and friends became available in 1995. Trying to share flags with the original program you're replacing is a noble idea. Unfortunately, there is a single flag that is in conflict between ssh and rcp -- -p. Because scp uses all the same connection rules (user, password, keypair) as ssh, I think it makes more sense to be consistent with ssh -- not rcp, which as of this writing, has been obsoleted for 15 years. As an end user, I shouldn't have to remember that ssh needs a -p to set a remote port, and scp needs a -P. Since ssh doesn't actually use -P for anything, that's where the original rcp "preserve modes" should land. <h2>copying local files</h2> scp is for copying files across a network. I don't want it to ever make local copies. If I want a local copy, I'll use cp, thank you very much. How many times have you accidentally omitted the trailing colon on a remote destination, just to end up with a user+host named local copy? <pre><code>$ touch testfile $ scp testfile mahlon@martini.nu $ ls -l mahlon\@martini.nu -rw-r--r-- 1 mahlon users 0 May 12 07:33 mahlon@martini.nu $ rm mahlon\@martini.nu </code></pre> When would you -ever- want that behavior, and want to use scp to do it? Never, I say! Here's what I think is an improvement: <pre><code>$ touch testfile $ scp testfile mahlon@martini.nu Cowardly refusing to make a local copy of 'testfile'. $ ^D </code></pre> <h2>local files validity</h2> Another one when copying local files to remote destinations. scp currently goes through all the work of connecting to the remote host before bothering to check if the local file you're trying to copy even is available. <pre><code>$ scp -v doesnt_exist mahlon@martini.nu: debug1: Reading configuration data /etc/ssh/ssh_config debug1: Connecting to martini.nu [] port 22. debug1: Connection established. debug1: [...] debug1: Found key in /home/mahlon/.ssh/known_hosts:5 debug1: Authentications that can continue: publickey,keyboard-interactive debug1: Next authentication method: publickey debug1: Offering public key: /home/mahlon/.ssh/id_dsa debug1: Server accepts key: pkalg ssh-dss blen 433 debug1: Authentication succeeded (publickey). debug1: channel 0: new [client-session] debug1: Entering interactive session. debug1: Sending command: scp -v -t . doesnt_exist: No such file or directory debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.0 seconds debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0 debug1: Exit status 1 </code></pre> Why bother? <pre><code>$ scp -v doesnt_exist mahlon@martini.nu: doesnt_exist: No such file or directory </code></pre> Rather than helplessly complain about this stuff into the internet abyss, I went ahead and made a quick patch. You get get it <a href="http://www.martini.nu/misc/scp.patch">here</a>, if you're so inclined. FreeBSD 8 PXEBoot with Grub2 http://www.martini.nu/blog/2010/01/freebsd-pxe-grub.html I just spent much longer on this than I care to admit. Hopefully this post will save someone else the time, until the dudes working on the <a href="http://www.gnu.org/software/grub/grub-2.en.html">grub2</a> project fill their documentation out. Trying to PXEboot a recent FreeBSD environment using grub2pxe? All the docs out there for PXE+FreeBSD from early 2000 (or worse) and using BSD's 'pxeboot' binary? Can't find anything on the 'kfreebsd' magic of grub2? Here's the magic sauce. Note that I compressed the kernel, so it'll be much faster under tftp. <pre><code>menuentry "FreeBSD 8.0 x86" { echo "Fetching the kernel and UFS root." echo "This could take a few minutes, so hang tight... " kfreebsd /freebsd/x86_8.0/kernel/kernel.gz Dh kfreebsd_loadenv /freebsd/x86_8.0/device.hints kfreebsd_module /freebsd/x86_8.0/mfsroot.gz type=mfs_root set kFreeBSD.vfs.root.mountfrom=ufs:/dev/md0c echo "Aaaiiight!" sleep 2 } </code></pre> So far, so good. Reality check http://www.martini.nu/blog/2009/08/gadget.html As I'm sitting here listening to a co-worker complain about how much his new phone "sucks" compared to his old one, and remembering the stream of issues he had with that one at the time, it slowly dawns on me how radically we've lost our sense of perspective. You've got a device that talks to satellites in space. It can talk to your computer network. You can communicate either via voice or text with anyone in the world, without wires. You can access the entirety of human knowledge (or at least what made it to the Internet) from your pocket. Take pictures. Videos. Share them with your family. Schedule meetings. Get directions, traffic status, and transit times from where you are standing to anywhere. Listen to a large pile of high quality music. Shop. Play video games. Track packages. Order movies to your house. Detect and tell you what constellation you're looking at (really.) It even (gasp) has an address book. The craziest thing though? The entire thing cost you about 200 dollars. This sort of miracle device was nonexistent 5 short years ago, and frankly, it is FUCKING MAGICAL. Have we always been like this? Or is it just as technology keeps improving by leaps and bounds our expectations rise in step? Did our sense of entitlement suddenly just skyrocket past what vendors can deliver in a pocket sized widget? If it doesn't respond instantaneously every time you swipe a finger across the screen, don't complain. Instead, take a moment and remember what age you're living in. We didn't have light bulbs 130 years ago. How is it that you can be so lucky?! Now shut up. I must just have a bootleggy face http://www.martini.nu/blog/2009/08/bootlegger.html Nadine and I trade each other a night a week for doing whatever. She usually uses it for getting her craft on, or having a cocktail with a pal. I use it to load up on happy hour margaritas and a movie with <a href="http://www.deveiate.org/">Michael</a>. We've been going to the same theater every week for about a year and a half now, usually directly from work. Unless there is nothing left to select from, we usually go even if there isn't a movie we're aching to see -- it's just fun to go to a nice theater, and ripping apart a bad movie can sometimes be as enjoyable as watching a good one. We're programmers for <a href="http://www.laika.com/">LAIKA</a>, and as such, we carry around company laptops. Going to a movie from work means we'll have laptop bags with us. Stressing this point: these aren't our laptops. They belong to LAIKA, and they contain keys into the internal network, along with various bits of intellectual property and privileged data. So if I go to the theater, and as I walk in I'm quietly pulled aside and asked to turn my bag in to a 16 year old minimum wage theater worker for "safe keeping", my answer will be an emphatic no. My answer earned us a conversation with the <a href="http://www.regmovies.com/">Regal</a> manager on duty last week, who hard-lined us for a solid 5 minutes before ultimately relenting, and offering us some comment cards to air our grievances. More likely than not, they'll be filtered into the trash can. She also apparently placed us on some sort of Regal movies terrorist list, which we realized this week as we returned to the theater (laptops in tow), and were 'identified' immediately after our ticket purchase. We were asked to stop for a chat with the <a href="http://www.bridgeport-village.com/movies.html">BridgePort Regal</a> General Manager. As we calmly refuted each assertion she made regarding Regal's <a href="http://www.regmovies.com/admittance-procedures.aspx">policy</a>, she too relented and let us into our movie. She quickly assumed a tactic of "it's not MY policy, I'm just doing what I'm told", which we were sympathetic to. I don't want to give the false impression that we hold any sway because we work at a movie studio, but I suggested that we could open a dialog with Regal corporate to help adjust their policy moving forward. <a href="http://coraline.com/">Coraline</a> was pirated too, after all. Of course, Coraline wasn't leaked from a theater at all, but rather the DVD distributor. Presumably, that's because of all the hard work the theaters are doing to prevent bootlegging. I'll save you the mundane play-by-play of our conversation, but here are the highlights. <ul> <li>"We don't allow bags into the theater." (As two women walked by us with purses.)</li> <li>"Well, it's not bags per se, but electronic devices." (As someone across the hall is talking loudly on his cell.)</li> <li>"Can you leave your bags at home?" (We go straight from work, and it would be a 35 mile trip for Michael to attend their theater if he went home first.)</li> <li>"Can you leave your bags in the car?" (Can you guarantee they'll be there when we return? Of course not. Not to mention the laptop is worth -more- than my car.)</li> </ul> I'm not blaming Kim (the General Manager) personally. She was, after all, just doing what she was told. Though she didn't give us a direct contact to have this conversation with for Regal corporate, she seemed a reasonable person. So what's a reasonable compromise here? We've spent a good deal of money on tickets in the last year and a half, and I don't want to feel discriminated against because I have what appears to be a laptop. I also don't want to stop going to the movies after work. How can we avoid having this conversation every week? How can Regal feel they are doing their part to stop movie piracy and not giving us special treatment, while we're simultaneously not punished for being a loyal patrons that happen to own some technology? Short term? Take any one of your workers. Let's be generous and say they make $10/hr. Have them come into the theater during the movie, and see if anyone is bootlegging! If you see me with my laptop open during the film, turned away from me, illuminating the chairs in front of me... oh man, yeah. Kick me the hell out. Ban me. Call the police. Even better because you've made an example out of someone for other would-be-bootleggers. You have 18 theaters, so worst case scenario when all of them are showing movies -- if it takes 3 minutes to do a sweep per theater -- will cost you about $9 for a full check. I'm fairly certain that will be significantly cheaper than confiscating all bags and/or electronic equipment that flow through your doors. You aren't a coat check, you aren't handing out ID tags for which device belongs to which person... because that's unrealistic and expensive. You already disallow cell phones from being used during the movie. It isn't unreasonable to extend that to all electronic devices, and then enforce it. That pretty well solves the problem completely and cheaply. It's also fair. I don't want to watch a film while some douchebag next to me is text messaging, or has an open laptop on Facebook. (I've never actually witnessed the latter.) Don't search people and don't confiscate. Just ask that when the lights do down, the devices turn off. Long term? Fix your policy. It is aged, and when it punishes regular customers instead of catching real problems, it's time to change it. It makes Regal look stupid when they honestly believe that a laptop is a device capable of making a decent bootleg. <pre><code>Film Theft: No recording devices (cameras, video recorders, sound recorders, etc.) are permitted to be used within any Regal Entertainment Group facility. Personal Communication Devices: In consideration of all our guests, Regal Entertainment group asks that you please refrain from using cell phones, pagers, or any other personal communication device while in the auditorium. </code></pre> The real problem is that they are making a distinction between devices that "record" and devices that "communicate". I've got news for you, Regal. They are one and the same. That iPhone is a better recording device than my laptop. Real camcorders are smaller than my hand and fit in my pocket! You're living in the future! Welcome to the age of tomorrow! For anyone that has seen "telesync" downloadable movies (aka: "cam" movies, named for the fact that they are recorded with a camcorder), I think we can all agree that they are generally -horrible- quality. Once in awhile, one looks halfway decent though. The sound is fairly clean. There isn't coughing, no jerking back and forth, and it is absent of any focus problems. Guess why? It was recorded from the projectionist booth, not the audience. For an audience member to make a copy that wouldn't be utter garbage, they'd need to set up in the center of the theater. With a tripod. And a good camcorder. And not get caught. Fortunately, they'd be caught when that employee came through to check during the film... right? The only significant source of piracy equating to tangible loss of revenue isn't shitty cam movies causing moviegoers to stay home. It's all the hands that DVD creation and distribution go through, causing potential DVD buyers not to purchase. To reiterate: piracy affects DVD sales, NOT box office. Then again, maybe I can put up with a horrible telesync copy of a movie, if it means I can watch it without being being forced into a weekly debate. Sales tips for the creepy http://www.martini.nu/blog/2008/12/sales.html I've been having a lot of door-to-door salespeople visit my house lately, and they are exhibiting some rather disturbing tactics. The following is a very short guide that should help a prospective sales dude not get punched in the teeth while visiting my house. 1) Don't use the "shave and a haircut" knock pattern That's a knock of familiarity. We're not buddies. I'm not going to welcome you to my porch with a warm hug. In fact, if you start out with a knock like that, you're immediately making me feel as if I'm being tricked into opening the door. 2) Don't just say your first name when I ask who's there So, now you've knocked, and I ask you through the door: "Who is it?" Please, say the name of your company -and- what you're doing on my porch. Don't respond with "David." Inevitably, you'll just be asked "David who?", and then "What do you want, David?" Again, it seems as if you're trying to trick me into opening the door, and worse, you're wasting my time while we play 20 questions. 3) Don't say you're doing "word of mouth" advertising It's not word of mouth if you're going door to door. For the record, that's called 'door to door'. Word of mouth means that your customers are doing your advertising for you. If you're not getting paid by a company, and you've discovered an AWESOME product that you just MUST go walking around the neighborhood to tell everyone you see about... okay then. You're allowed to call it word of mouth. 4) Don't say you aren't selling anything If I tell you "I'm not interested", don't reply indignantly with "Wait, but I'm not selling anything!!" If you're advertising on my doorstep, you're selling something, even if I'm not directly giving you money at the moment. That's the definition of advertising. You or your employer has a product that they want me to have, in exchange for my money. If that indeed is the case and you really don't have anything to sell me, that just means you are a transient and have even less reason to be hanging out at my house. Please leave. 5) Don't compliment me on my children Telling me how beautiful my little girls are does NOT endear me to you. It instantly brings forth a feeling of parental terror that cascades over me, as I realize a stranger (that knows where I live) finds my children attractive. This makes you appear to be predatory in a really bad way, no matter how innocent the comment actually might be. In any event, I certainly won't be overcome with an intense impulse purchasing desire. Eh, maybe it's just time to invest in a tacky little 'no soliciting' sign. Or a big dog. RubyConf 2008! http://www.martini.nu/blog/2008/11/mobile_1.html <a href="/images/blog/2008/11/1225979863656.jpg" class="highslide" onclick="return hs.expand(this)"> <img src="/images/blog/2008/11/thumb_1225979863656.jpg" alt="mobile post image" /> </a> Way more crowded than I was expecting. Total sausage fest. I think I saw one girl, and she was being mobbed by droves of adoring programmers. Tell PF to fix PS3 fragmented packets http://www.martini.nu/blog/2008/10/ps3-pf.html Here's a quickie for anyone using a <a href="http://www.openbsd.org/faq/pf/">pf</a> on their home router, and having problems with multiplayer on a <a href="http://www.us.playstation.com/PS3">ps3</a> inside the network. Apparently, I wasn't the <a href="http://www.google.com/search?q=ps3+fragment+nat">only one</a>. The PlayStation appears to perform packet fragmentation while leaving the don't-fragment bit set. Most home routers don't seem to care -- but there are some that catch this strange behavior and not allow it to pass. pf is generally strict about this sort of thing if you have scrubbing enabled (which is always a good idea.) Here's a workaround to get you gaming again. <pre><code>scrub from [ps3-ip-address] to any no-df random-id fragment reassemble </code></pre> The only required flag here is the no-df, which allows a packet through that would normally be dropped. pf recommends also ensuring IP identifier uniqueness, which the random-id flag enables. I also re-assemble the packets before forwarding them upstream, though this is entirely optional. The nice part about this is that it only fixes up the stuff coming from the PS3 -- you can still have the rest of your scrub rules cleaning up nasty goop coming into your network as normal. So far, the only game that is still giving me trouble is GTA4. I successfully join a game about 1 in 4 tries. Every other game I've tried has been absolutely fine.