Lisp/Scheme/Emacs Pipe

The Zipper [PDF]

Sat, 21 Nov 2009 06:51:45 -0800

submitted by lispnik
[link] [14 comments]

Zach Beane: Go try XONG

Sat, 21 Nov 2009 12:09:34 -0800

David O'Toole put out version 1.0 of his roguelike puzzle game XONG yesterday.

When he asked me to try it out, I expected some headache when downloading it, compiling it, grabbing libraries, etc. Nope! I'm using Snow Leopard on an Intel MacBook. I grabbed the .dmg, double-clicked the xong icon, and it ran perfectly. It bundles an executable SBCL image together with the graphics, sounds, and libraries it needs, so it's completely self-contained and looks like any other Mac .app bundle. It's nice to see this kind of application delivery.

Lispjobs: Software/firmware developer (Lisp), San Diego, CA

Sat, 21 Nov 2009 08:18:09 -0800

For biomedical devices, Software/Firmware Developer – Lisp (Contract to Hire).

Zach Beane: LISP is difficult to learn

Fri, 20 Nov 2009 15:05:35 -0800

The local grocery store has a big bin of used books for $1. After seeing a 1983 computer dictionary's description of LISP, I knew I had to buy it:

It says:

LISP acronym for LIST PROCESSING a high-level programming language use primarily for list processing, symbol manipulation, and recursive operations: it can handle many different data types, treat programs as data, and provide for the self-modification of the program as it is executing: generally considered a difficult language to learn

(Emphasis mine.)

François-René Rideau: Software Irresponsibility

Fri, 20 Nov 2009 06:05:02 -0800

In the way of achieving a healthy software development environment, a lot of projects fall in one of these two DON'Ts: irresponsibility and territoriality. Irresponsibility is when there is no one in charge of making things right (with respect to a whole category of problems). Territoriality is when there is someone in charge, and he won't let anyone else touch the code without him. They may sound opposite to one another, but often irresponsibility is a result of territoriality, where the person in charge just isn't interested in the kind of problem you're experiencing, and so any consideration for such problems gets disregarded in favor of whatever fits the interests of the maintainer.

As a notorious example of irresponsible software design, I will cite PHP, whose maintainer has repeatedly claimed he was never interested in designing a good programming language. It shows. But I'm sure you'll find plenty of familiar examples in your own experience, from the irresponsive provider of some proprietary software you've been locked into using, to maybe someone from some other department of your company that you've had to deal with, and perhaps even to yourself with respect to your own users.

Of course, it is always possible to start a new project to replace the software you're dissatisfied with. But the costs associated to such endeavour are often huge. You will have to reimplement not just the parts that the replaced software did wrong, but also the parts it did right. While by assumption you can clearly see parts of the design that are wrong and do better with respect to those parts, you may be lacking the proficiency, time and interest to do as well on these other parts. Last but not least, the community around the original software, that made it valuable, will not switch to the new one for merely small gains, and the transition from the old system to the new system may impose demanding constraints of backwards compatibility.

Free Software can drastically reduce the cost of starting a new project, by allowing you to fork the existing code base, copy large chunks of it, or at the very least read the code of what you may have to be compatible with. It does make the cost go down to zero, however. There again, cruft that made the original software bad enough that you felt like changing it will be a heavy burden upon you if you keep it, and a huge setback to reimplement if you decide to throw it away. And even in the world of free software, some licenses can incur a higher cost to forking than others: for instance, licenses that force you to distribute the code as source, that prevent you from making any adaptation and keep the name, or otherwise put hurdles in the way of forkers.

In the world of Common Lisp, an interesting case of not-so-responsible software is ASDF. Yeah, some of you might be tempted to sneer, is he going to blame the problem with ASDF on the original author who is long gone, and the current maintainer who isn't committing enough energy and availability to making the project go forward? No, not at all. Dan Barlow actually did a great job while he was there, and if no one is stepping up to offer more than Gary King provides, it's certainly not Gary who's to blame; he is getting blatant bugs fixed and accepting consensual patches, and at this point forking would be trivial to fork the git repo if anyone were dissatisfied with Gary's performance. Rather, I'll argue that any problem with ASDF is a symptom of the way responsibilities are distributed with respect to its update, and propose a small technical change that can hopefully improve things a lot.

When Dan Barlow wrote ASDF in 2001, he only wanted a better CL build system that he could use on top of SBCL for his own projects, something that would be more declarative than the previous best alternative, MK:DEFSYSTEM, and would only need to support modern ANSI CL implementations, not old forgotten Lisp variants. In developing ASDF, he loosely followed the better design principles once proposed by Kent Pitman for an extensible CL build system. In all those objectives, ASDF was a tremendous success, and CL users should all be grateful to Dan was his achievement. However, as usual the problem lies with issues he didn't try to tackle.

Problems with ASDF include the painful way you have to configure it, deficient support for documentation or testing, broken support for conditional compilation, missing support for generated files, its weird TRAVERSE algorithm, etc. Unlike the issue of deterministic incremental compilation that prompted me to start XCVB, none of the above issues is unsolvable in itself. But the constraints within which ASDF is developed make it hard to solve them. Happily, some of these constraints can be removed with a little bit of hackery. Unhappily, others are intrinsic to the choice of constraints.

Indeed, ASDF has a distribution bootstrap problem: it specifies how to load other systems, but nothing specifies how ASDF itself is loaded into your system. That problem is left to whoever writes the Lisp implementation, distributes the Lisp package, or maintains a larger project within which ASDF is used. And in the world of Common Lisp, there are tens of different implementations, many of them packaged in various ways. To make the problem tractable, ASDF follows the constraint that it comes as a single file that may be loaded independently of its location, and doesn't depend on any third party library beside what is included by each supported implementation.

This means that the provider of libraries that use ASDF must rely on the least common denominator of ASDF features, unless he wants to force his users to make sure they upgrade their ASDF. Upgrading the current ASDF could conceivably be easy, but is actually hard, because of the inability to hot-upgrade a loaded ASDF. It is still possible to remove an old ASDF and replace it, and XCVB does it in an ugly way (see how xcvb/no-asdf deletes the ASDF package and anything that uses it); however, files compiled against the old ASDF may not work with the new one and vice versa (I've had "interesting" issues that way while using xcvb-master in slightly different setups).

This inability to upgrade further makes the pre-packaging of a Lisp implementation with a given version ASDF a bit of a damned if you do, damned if you don't proposition to Lisp implementers and distributors. Happily, implementers have the option of only providing ASDF when REQUIRE'd, and this is the preferred way for users to specify they want to use ASDF, when it is available. Distributors may not have this option inasmuch as there is no standard mechanism to hook into an implementation's REQUIRE. An example of a failed attempt at distributing Common Lisp with ASDF is common-lisp-controller. It tried to provide a way to make sure there is always a configured ASDF, but their solution only works if everyone uses only debian or another supported OS distribution, and only if the debian maintainers keep up with all the common-lisp software (implementations and libraries) that one may want to use; two hypotheses that have been disproved in practice with respect to the needs of many users.

To step back to the problem of responsibility, the current constraints of ASDF are such that a consistent upgrade of ASDF requires action on the part of tens of people, each in his own territory: ASDF contributors, Lisp implementers, package distributors. If only one hacker alone decides to upgrade ASDF, he has to incur the cost of the associated work of modifying ASDF and the risk that this will introduce incompatibilities, for little overall progress in terms of what users may rely upon unless he convinces tens of other people also do the same. Thus, the costs of adoption are high, the benefits of development are low, development is slow, and a cycle of stagnation continues.

If someone cares about the future of ASDF, I recommend they should address the upgradability bug, which will itself unlock a big hurdle on the way of further evolution of ASDF. Note that this upgradability includes both the technical ability to load a new ASDF on top of an old one, but probably also the proper configuration of ASDF, to load systems from a series of configured paths declared in user and system preference files.

Personally, I'd rather work on XCVB, and invite people interested in build systems to work on it instead of ASDF. But if XCVB is to prevail, it may as well prevail against the best ASDF that ASDF can be.

XCVB tries to avoid all the problems of ASDF by stepping back from the assumption that everything happens in a one Lisp world that has already been setup. Instead, XCVB will manage many Lisp worlds that it will help you setup. It doesn't live inside your Lisp image, and thus doesn't depend on Lisp implementers or distributors to provide it for you; it can therefore evolve fast without any issue due to slow distribution. It is also liberated from the constraint ASDF has of itself having no dependencies, because these dependencies would both make bootstrap more difficult and interfere with other versions of same software as part of the software being built. XCVB thus reuses plenty of libraries and is richer in features, and will use even more libraries in the future. All this in addition to its original essential distinguishing feature of providing deterministic incremental compilation.

XCVB isn't complete yet, but I believe that on supported implementations (currently only SBCL, CCL, CLISP, but I could add more on demand, or you could do it yourself), it is already better than ASDF. If you care for CL build systems, please fix ASDF; but please give XCVB a try, too.

Even if you don't care for CL build systems, organize your software developments so that every upgrade step can be done by the single maintainer of a single project, without requiring the coordination between many people in many different projects.

This lesson should be particularly dear to those who dream of making a Standard for Lisp, Scheme, Javascript, Java or any other language or piece of software: if you need to agree on something that will require future coordination between plenty of people, it is almost sure to fail to evolve fast enough to remain relevant, even though one or two iterations might make it live (see e.g. R6RS and its current attempted successor). Instead, you should offer software that already runs with a free software reference implementation that others can trivially adapt to their systems if needed (also in Scheme, compare the SRFI process). After your code has become de facto standard, you might not even care for it being any kind of de jure standard anymore. Whereas if there's going to be a de jure standard, it will be blood and conflict as long as there's anything to disagree about, and still resentment at the unsatisfying compromise that results.

The same idea also accounts for Conway's Law: if you divide your software project between many teams, each team will build its own code base, and the structure of the software will end up following the structure of the teams that build it. Parts of the software where the ownership is not clearly assigned but where authority is divided will be the scene for conflict, slow evolution, bad design (if design at all), etc. Vertical disintegration, where independent teams manage layers of the software and its life-cycle, will in particular lead to a multiplication of such dysfunctional interfaces between entrenched overlapping code bases the global architecture of which cannot be refactored.

Happily, solving the problem of irresponsible software is much easier than solving the problem of irresponsible government.

Vladimir Sedach: November Montreal Scheme/Lisp User Group presentation

Thu, 19 Nov 2009 13:36:00 -0800

On November 17, I gave a presentation to the Montreal Scheme/Lisp User Group about developing high-performance network servers in Lisp. I looked at the lessons to be learned from existing Common Lisp web servers, notably Antiweb and TPD2, tried out some of the ideas in my own new under-development web server HTTP DOHC along with some new ideas about threading and concurrency, and put the results of my findings into the presentation.

If you're interested in the topic, you should also watch John Fremlin's TPD2 presentation to the Shibuya Lisp user's group.

I plan on making HTTP DOHC a full-featured Common Lisp web server with an interface that's somewhat compatible with Hunchentoot 1.0.

In other news, I now have time available for contract work. Get in touch at vsedach@gmail.com.

Patrick Stein: Playing with Sheeple-based GUI atop CL-OpenGL

Thu, 19 Nov 2009 08:59:55 -0800

Inspired by the previous TC Lispers meeting and spurred on by the probable topic of the next TC Lispers meeting, I have spent the little bit of coding time I’ve had over the past two weeks on making a GUI layer using Sheeple atop CL-OpenGL and employing ZPB-TTF for font-loading.

I have dubbed this project Woolly (because it’s made from Sheeple and because Woolly sounds sorta like GUI).

I spent about half of my coding time so far getting the basic framework in place (proven through clickable buttons with labels). I am trying to keep it cleanly separated between generic GUI stuff and the CL-OpenGL specifics in the event that someone would like to port it to some other I/O spec.

The other half of my coding time was spent getting the font-rendering to be anti-aliased. I promise to write more about how I accomplished the font-rendering in a future post so that if you’re ever stuck rendering fonts in OpenGL, you won’t be stuck with pixelated blockiness or resorting to rendering to a texture-map and letting the mipmapper figure it out.

For this post, however, I’ll just show you the code that sets up the interface depicted here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

(require :asdf)
(asdf:operate 'asdf:load-op 'woolly-gl :verbose nil)

;; make it easier to change renderer/controller later
(rename-package 'woolly-gl 'toolkit)

(defun test ()
(let ((font (sheeple:object :parents toolkit:=font=
:em-size 48
:pathname "okolaks/okolaksRegular.ttf")))
(let ((app (sheeple:object :parents toolkit:=app=))
(win (sheeple:object :parents toolkit:=window=
:title "Woolly Window 1"
:width 640
:height 480))
(but (sheeple:object :parents toolkit:=button=
:offset-x 40
:offset-y 40
:width 300
:height 100
:font font
:label "Button")))
(woolly:display-window win)
(woolly:add win but)
(woolly:main-loop app)
(woolly:destroy-window win))))

Next up on my agenda is to make the background and button prettier. It should be easy enough to do with GL_LIGHTING and some vertex-coloring for gradations. After that, it’s on to more controls like labels, panels, checkboxes, drop-downs, borders, and (my dread) text input boxes. Then, it’s on to a layout manager.

Edit: Here’s the same GUI a day later. I’m using a simple lighting scheme and rendering the button in 3D. I haven’t yet hooked in the bit to render it depressed when the button is pressed. I’ve tested the code that draws it the other way, but I haven’t hooked it into the mouse handlers yet.

Edit #2: Actually, it only took a few minutes for me to hook in the rendering it pressed vs. unpressed. When I did it though, it looked like the label was sliding around because the effect of the contrast between the light and dark edges of the button was so great that you perceive the whole button sliding when it’s pressed. So, I added a little bit in there to actually slide the text by an amount close to what is perceived. So, now… it looks pretty spiffy.

Christophe Rhodes: 19 Nov 2009

Thu, 19 Nov 2009 05:24:17 -0800

A while ago now, Nikodemus Siivola moved bug information for SBCL from a flat text file to Launchpad. Historically I have had almost nothing but displeasure working with the "standard" or "industrial" bug trackers; I have found Bugzilla horrible to work with, both as a bug reporter and as an administrator; lighter-weight solutions such as Trac are just about tolerable, but basically anything that requires me to have a Web Browser seems to end up confusing and distracting me. An honourable mention at this point goes to debbugs: being able to report, manipulate, update and close bugs by e-mail is close to my idea of Nirvana.

So, Launchpad. Initially, I was dismayed, because there doesn't seem to be a way of getting notifications of bug updates over RSS, which would be a second-best to getting updates by e-mail. I managed to ignore all SBCL bug reports for a while, but eventually I bit the bullet and signed up (having refused to do so a good long while ago, when shortly after I used Ubuntu's bugzilla to report a bug they closed the bugzilla in favour of launchpad without managing to transfer accounts across.)

A large motivation for signing up was the discovery that launchpad does, in fact, have an email interface to the bug tracker; as long as you can emit GPG-signed mail (which I can), it seems to have all the required functionality for doing things without needing to go near a web browser; I can now receive bug reports and reply to them, and in at least some cases the References: headers in the mail I receive allows my client to thread the discussion properly (I haven't really stress-tested this yet, but it works at least well enough for now.)

If that were all, this would not be news (and not even worthy of a blog post). But now I get to demonstrate my Emacs lisp “scripting” ability, in much the same way as Dan Barlow did for me many years ago: SBCL has a mailing list for reporting bugs, for people who are unsure as to whether their problem is a bug or not, or for people who don't want to go to the trouble to get a launchpad account just to report a bug. When such a report does describe a new bug that we should be tracking, that report needs to make its way to launchpad.

Without too much further ado, I present sbcl-bugs-mail-forward, which constructs a message (almost) ready to be sent:

(defun sbcl-bugs-mail-forward () (interactive) (let ((message-forward-ignored-headers "") from subject) (gnus-summary-mail-forward 4) (message-goto-to) (insert "new@bugs.launchpad.net") (message-goto-subject) (message-beginning-of-line) (re-search-forward "\\[\\(.*\\)\\].*\\[\\(.*\\)\\] \\(.*\\)$") (setq from (match-string 1) subject (match-string 3)) (message-beginning-of-line) (let ((kill-whole-line nil)) (kill-line)) (insert subject) (message-goto-body) (insert "Report from " from "\n\n") (insert " affects sbcl\n status confirmed\n importance ") (save-excursion (insert "\n tag \n done\n\n") (message-goto-body) (re-search-forward "^\\(-\\)+ Start of forwarded message \\(-\\)+$") (beginning-of-line) (let ((kill-whole-line t)) (kill-line)) (re-search-forward "^\\(-\\)+$") (beginning-of-line) (end-of-buffer) (kill-region (mark) (point))) (mml-secure-message-sign-pgpmime)))

You can tell it's scripting, really: it's an odd mixture of plausible and dubious ways of getting things done: regular expressions to extract the original sender of the report, and to remove the forwarded message information (and the dull advert inserted in the footer by SourceForge's mailing list system). On the other hand, that function, coupled with something along the lines of

(setq gnus-parameters '(("nnml\\+private:list.sbcl-bugs" (gnus-summary-prepared-hook '(lambda () (local-set-key (kbd "C-c C-f") 'sbcl-bugs-mail-forward) (local-set-key (kbd "S o m") 'sbcl-bugs-mail-forward))))))

gives me exactly what I think I want: a simple way of creating, tagging and classifying an entry in the bug tracker from a mail report.

I couldn't find any convenient emacs/launchpad interfaces (or any at all, in fact); I'm not sure this counts as one either, but by all means use, adapt and improve on the above for your purposes – I'll happily take criticism of and improvements to this hack.

Vladimir Sedach: Templates and code generation

Wed, 18 Nov 2009 14:12:00 -0800

Earlier this month Google released Closure Templates, a new templating library intended for generating HTML. Who cares? I personally dislike HTML templating, and avoid it whenever possible in lieu of s-expression based generation tools like CL-WHO. At first look, Closure Templates didn't seem to be anything new or useful.

The one thing that makes Closure Templates somewhat interesting is that the library works in both Java and JavaScript, so you get client and server-side templates in one place. I did a similar thing with uri-template by having the URI template expand into code that executed in both Common Lisp and JavaScript via Parenscript (here I have to state that despite being an author of a templating library, I still dislike templating libraries and even find my own creation annoying at times).

About a week after Closure Templates was released, Andrey Moskvitin (archimag) wrote an implementation in Common Lisp (interesting note: it took him only five days and 1/15 the number of lines of code as the original). The resulting system, cl-closure-template, similarly generates JavaScript via Parenscript.

I still didn't understand the motivation. Yesterday, Andrey was kind enough to explain it. Those pampered by web development in Common Lisp using s-expression HTML generation tools simply don't encounter the problem of trying to fit HTML templating onto the paradigm of incremental page updating via AJAX. So if you want to build an AJAX web application and use HTML templating, give cl-closure-template a try.

[Blog metanote: if you're reading this blog through its feed, you will shortly see this post show up in Russian. a CONS is an object which cares is now syndicated on Russian Lisp Planet, and I'm going to be writing Lisp-related posts in Russian as well as English. All such posts will be tagged 'lisp-ru'. Filter out that tag if you don't want the Russian version of the posts to show up in your feed reader.]

Christophe Rhodes: 18 Nov 2009

Wed, 18 Nov 2009 04:02:50 -0800

As mentioned earlier: the 2010 European Lisp Symposium invites your contributions. Unfortunately, the website for the 2010 event is not set up yet; you can get an impression of what the event is like by looking at last year's website, which in the fullness of time (soon, I hope) will be updated with ELS2010 information. In the meantime, here's the Call for Contributions: we would welcome both papers describing original work, not published elsewhere, and submissions for tutorial sessions. Submission will be through EasyChair's conference management system.

3rd European Lisp Symposium

May 6-7, 2010, Fundação Calouste Gulbenkian, Lisbon, Portugal

Important Dates

Submission Deadline: January 29, 2010
Author Notification: March 1, 2010
Final Paper Due: March 26, 2010
Symposium: May 6-7, 2010

Authors of accepted research contributions will be invited to submit an extended version of their papers to a special issue of the Journal of Universal Computer Science (J.UCS).

Scope

The purpose of the European Lisp Symposium is to provide a forum for the discussion and dissemination of all aspects of design, implementation and application of any of the Lisp dialects. We encourage everyone interested in Lisp to participate.

The European Lisp Symposium 2010 invites high quality papers about novel research results, insights and lessons learned from practical applications, and educational perspectives, all involving Lisp dialects, including Common Lisp, Scheme, Emacs Lisp, AutoLisp, ISLISP, Dylan, Clojure, and so on.

Topics include, but are not limited to:

Language design and implementation
Language integration, interoperation and deployment
Development methodologies, support and environments
Reflection, protocols and meta-level architectures
Lisp in Education
Parallel, distributed and scientific computing
Large and ultra-large-scale systems
Hardware, virtual machine and embedded applications
Domain-oriented programming
Lisp pearls
Experience reports and case studies

We invite submissions in two categories: original contributions and tutorials.

Original contributions should neither have been published previously nor be under review in any other refereed events or publication. Research papers should describe work that advances the current state of the art, or presents old results from a new perspective. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The programme committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality.
Tutorial submissions should be extended abstracts of up to four pages for in-depth presentations about topics of special interest for at least 90 minutes and up to 180 minutes. The programme committee will evaluate tutorial proposals based on the likely interest in the topic matter, the clarity of the presentation in the extended abstract, and the scope for interactive participation.

The tutorials will run during the symposium on May 6, 2010.

Programme Chair

Christophe Rhodes, Goldsmiths, University of London, UK

Local Chair

António Leitão, Technical University of Lisbon, Portugal

Programme Committee

Marco Antoniotti, Università Milano Bicocca, Italy
Giuseppe Attardi, Università di Pisa, Italy
Pascal Costanza, Vrije Universiteit Brussel, Belgium
Irène Anne Durand, Université Bordeaux I, France
Marc Feeley, Université de Montréal, Canada
Ron Garret, Amalgamated Widgets Unlimited, USA
Gregor Kiczales, University of British Columbia, Canada
Nick Levine, Ravenbrook Ltd, UK
Scott McKay, ITA Software, Inc., USA
Peter Norvig, Google Inc., USA
Kent Pitman, PTC, USA
Christian Queinnec, Université Pierre et Marie Curie, France
Robert Strandh, Université Bordeaux I, France
Didier Verna, EPITA Research and Development Laboratory, France
Barry Wilkes, Citi, UK
Taiichi Yuasa, Kyoto University, Japan

Andreas Fuchs: Hunchentoot gets a debugging-acceptor

Sun, 15 Nov 2009 13:11:36 -0800

Today, I submitted a patch (the first free software lisp one in months for me!) to the Hunchentoot project, and it got accepted. Yay!

Some backstory: Hunchentoot’s 1.0.0 release dropped a lot of implementation-dependent features, among them functionality to invoke the debugger if an error happens while handling a request. While workarounds exist, none of them were obvious to new users or users who recently upgraded.

The patch I sent should fix this, hopefully. It adds a rudimentary error handling protocol to Hunchentoot, and provides two generic functions whose behavior can be adapted to your error handling needs. You can see for yourself in Hunchentoot’s svn repository.

If you’re a Hunchentoot user, I urge you to test this (in both development mode using debuggable-acceptor and running with the default settings). The sooner you find bugs, the sooner they can be fixed, the sooner a release can be pushed out. And if you don’t find bugs at all, that’s cool, too (-:

Grant Rettke: Carl Sagan’s Apple Pie

Grant — Sat, 21 Nov 2009 08:38:08 -0800

I heard this joke a long time ago and somehow it never gets old, especially when the person telling the joke turns things around a bit.

His recipe is a classic; his sense of humor is hard to resist:

(image found via Neven)

Grant Rettke: Conway’s Law

Grant — Sat, 21 Nov 2009 06:55:03 -0800

Conway’s Law is not intended to be a joke or nasty witticism:

…organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.

– Melvin Conway

(via Fare via Wikipedia)

Programming Praxis: Master Mind, Part 2

programmingpraxis — Fri, 20 Nov 2009 01:00:49 -0800

In the previous exercise we wrote a Master Mind setter. In today’s exercise we will write a solver, using an algorithm given by Donald E. Knuth in his article “The Computer As Master Mind” in Volume 9, Number 1, the 1976-1977 edition, of The Journal of Recreational Mathematics.

The central concept of Knuth’s algorithm is the pool of potential solutions. His algorithm chooses at each step a probe that minimizes the maximum number of remaining possibilities over all possible response of the codebreaker; in the event of a tie, any pattern that achieves the minimum may be used, subject to the condition that a probe that is a member of the current pool is preferred to one that is not.

For instance, consider the pool of 1296 possible code words at the start of a puzzle. There are essentially five possible starting probes: 1 1 1 1, 1 1 1 2, 1 1 2 2, 1 1 2 3, and 1 2 3 4 (rotations are excluded, as are variants that substitute one symbol consistently for another). The remaining pool sizes after each of the five probes is applied to all of the 1296 possible code words are given in the table below:

	1111	1112	1122	1123	1234
`....`	625	256	256	81	16
`W...`	0	308	256	276	152
`B...`	500	317	256	182	108
`WW..`	0	61	96	222	312
`BW..`	0	156	208	230	252
`BB..`	150	123	114	105	96
`WWW.`	0	0	16	44	136
`BWW.`	0	27	36	84	132
`BBW.`	0	24	32	40	48
`BBB.`	20	20	20	20	20
`WWWW`	0	0	1	2	9
`BWWW`	0	0	0	4	8
`BBWW`	0	3	4	5	6
`BBBB`	1	1	1	1	1
max	625	317	256	276	312

The minimax solution is 256, achieved when the probe is 1 1 2 2, so that should always be the first probe. Then the solution is determined by making the minimax probe, reducing the pool by applying the result of the probe, and repeating on the reduced pool until the puzzle is solved.

Your task is to write a Master Mind solver based on the rules set out in the previous exercise and Knuth’s algorithm given above. When you are finished, you are welcome to read or run a suggested solution, or to post your solution or discuss the exercise in the comments below.

Ben Simon: NuGram - an architecture worth pondering

Ben Simon (noreply@blogger.com) — Thu, 19 Nov 2009 07:26:43 -0800

Dominique Boucher was kind enough to document the architecture of the NuGram web service. It's an interesting mixture of Scheme and Erlang, and serves as an excellent reminder that there's more approaches to developing scalable software than just Apache + MVC (aka: PHP, Ruby, Java, etc.).

The part that really got my attention, though was:

Most requests to the RESTful API are done in the context of a session. Each session is associated with an Erlang process and the application keeps a mapping between the session ID and the process for the session in the Mnesia database (it is rather cool to store things like process IDs in a database!). So when the application receives a request, it extracts the session ID from the request URI, finds the corresponding process in the database, and simply forwards the request to that process.

Think about that: a request comes in, and you find the running process that is sitting around, ready to serve it. No process yet? Then create a fresh one. And when a session expires, you kill the process.

What a remarkable architecture (and so perfect for Erlang)! It provides an usual level of modularity, as you can effectively program a single process for a single interaction, and then let Erlang scale it all up.

From a scalability perspective, this is huge, as the system is designed from the ground up to work with individual processes - whether those processes are running on a single machine, or spread out among machines, I assume could be transparent.

As an experiment, it would be interesting to compare traditional session programming, versus 1-session-per-process (like above), versus web continuations. I wonder if one of those models is easier to program with than the other. And I wonder if one scales better than the other.

If nothing else, it makes for an excellent thought experiment.

Thanks Dominique!

Update: Dominique was kind enough to address the thoughts above on his blog. So check it out, he's got some excellent points.

Dominique Boucher: 1-session-per-process - some further comments on the NuGram architecture

Dominique Boucher (noreply@blogger.com) — Wed, 18 Nov 2009 18:08:57 -0800

In response to my previous post on the NuGram architecture, Ben Simon wrote a post in which he focused most exclusively on the idea of using a single process for handling all the requests for a given session. I wanted to add a few comments about this approach.

First, the idea is not new at all. In fact, I'd say that this is a pretty standard approach in Erlang and the language facilitates its implementation. For instance, the open-source ejabberd XMPP server uses this approach.

Also, it is true that it is completely transparent to the system whether the session's process runs in the same Erlang VM or on a remote machine. The syntax is exactly the same: Process ! Message. That's it. In the case of NuGram Server, the database (which replicates the session table on all nodes) holds references to Erlang processes. Once it has obtained the reference to the session's process, it simply sends a message encapsulating the request to that process (using the ! notation).

Of course, there are some complications if the node on which the process runs suddenly crashes or becomes unavailable. Session replication is less trivial to implement. In our case, the system maintains, together with the session table, a table holding all the relevant data to recreate a session if needed. This was fairly easy to do in our server given our requirements and the nature of the API. For more complex APIs, this could be much more challenging.

Although I'm a big fan of the 1-session-per-process approach (it is a very effective one for the implementation of comet-like servers), it has some limitations that the continuation-based approach do not suffer. The back-button/bookmarking problem immediately comes to mind. Serializable continuations can be put in a database for later retrieval. But the question remains whether this is a practical approach. For instance, for how long do you retain continuations in the database? What happens when the code changes?

Another benefit of using the continuation-based approach is the fact that the code handling the request is written in a more direct style. By this, I mean that the application flow is coded in a more sequential way: do this, then do that, etc. You don't have to code using state machines or callbacks.

But this can be achieved in Erlang as well using two processes per session. (Processes are so cheap in Erlang!) This approach can be used to implement dialog-based applications and providing the illusion of a synchronous API, à la Tropo). I'll talk more about this in another post very soon.

Grant Rettke: The Grass is Always Greener

Grant — Wed, 18 Nov 2009 13:33:32 -0800

Here is a little tongue in cheek developer career philosophy/humor:

(via xkcd via Greg Wilson)

Programming Praxis: Master Mind, Part 1

programmingpraxis — Tue, 17 Nov 2009 01:00:10 -0800

Master Mind is a two-player game of deductive logic. One player, the setter, selects a four-symbol code, and the other player, the solver, tries to identify the code by trying test patterns, probes, to which the setter responds with the number of black hits, indicating the number of positions where the code symbol and probe symbol are identical, and the number of white hits, where a probe has the right symbol in the wrong position. Setter and solver change roles after each puzzle is solved, for a pre-defined number of rounds, and the winner is the player who has solved the puzzles with the least number of probes. With six symbols and four pegs there are 6⁴ = 1296 possible codes. The physical game uses colored pegs for the symbols; we will use digits instead. Variants of the game increase the number of symbols and/or the length of the code; a variant with five pegs using eight colors is marketed under the name Super Master Mind. Here is a sample game:

1 1 2 2 B 1 3 4 4 W 3 5 2 6 BWW 1 4 6 2 BW 3 6 3 2 BBBB

The solver first probes with the pattern 1 1 2 2, which has a single black hit. The second probe, 1 3 4 4, receives a single white hit. The third probe, 3 5 2 6, earns a single black hit and two white hits. The fourth probe, 1 4 6 2, receives one black hit and one white hit. The fifth probe, 3 6 3 2, solves the puzzle with four black hits.

Your task is to write a program that performs the role of the setter, selecting a random code, prompting for probes, and scoring each probe until the human solver who is playing the game solves the puzzle. In the next exercise you will be asked to write a solver. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

Llewellyn Pritchard: Experimental monolitic IronScheme executable

leppie — Mon, 16 Nov 2009 07:35:52 -0800

A common request is to have some form of compiled libraries to run a Scheme program, and not relying on source files for the program to run.

This is already provided in IronScheme via precompiled/serialized libraries, but still results in a ‘binary’ format for every library file, which in turn can end up being quite a few with a fixed directory structure.

So yesterday I worked around a serialization bug that hindered my progress on improving this.

I have added the following:
1. Created a container package to shove all precompiled libraries in.
2. Added the ability to use ILMerge to merge all the assemblies into 1 executable.
3. And for bonus points, I added the ability to include the container package into such a merged executable, resulting in 1 fat executable with all the libraries included.

For now, you can go to this IronScheme page and download the result of step 3. A 21MB executable with everything to run IronScheme, including all IronScheme and SRFI and miscellaneous libraries.

Of course your own program may not need all the libraries, so you could chose to only merge the ‘base’ 2.1mb executable with say 1mb of libraries.

I will add some documentation later to explain how to craft these.

Cheers

leppie

Yinso Chen: BZLIB/SESSION.plt - a Session Store via BZLIB/DBI.plt

yc (noreply@blogger.com) — Sun, 15 Nov 2009 22:32:24 -0800

I originally planned to write a series on the development of a session store, but it turned out that there aren't that many things to write about, so I am just going to release the code via planet. As usual, this is released under LGPL.

Installation

To download the planet package:


(require (planet bzlib/session))

The package comes with three separate database installation scripts (one each for sqlite, mysql, and postgresql). You can call them via the following:


;; installing to a sqlite database
(require (planet bzlib/session/setup/jsqlite)) (setup-session-store/jsqlite! <path-to-sqlite-db>) ;; installing to a mysql database (require (planet bzlib/session/setup/jazmysql)) (setup-session-store/jazmysql! <host> <port> <user> <password> <schema>) ;; installing to a postgresql database (require (planet bzlib/session/setup/spgsql)) (setup-session-store/spgsql! <host> <port> <user> <password> <database>)

Known Issue: the script currently can only be run once and it assumes the table session_t does not exist in the target database. This will be rectified in the future.

Session ID

The session store uses uuid for session IDs. bzlib/base provides API for manipulation of uuids.

To create an uuid, just run (make-uuid). It can optionally takes in a parameter that are either an uuid in string, and then create the corresponding uuid structure (it can also takes in another uuid structure and make an equivalent uuid structure).


> (require (planet bzlib/base))
> (make-uuid)
#<uuid:4ba52eac-a0b4-415a-88f5-57d1fadd1aba>
> (make-uuid "4ba52eac-a0b4-415a-88f5-57d1fadd1aba")
#<uuid:4ba52eac-a0b4-415a-88f5-57d1fadd1aba>
> (make-uuid (make-uuid "4ba52eac-a0b4-415a-88f5-57d1fadd1aba"))
#<uuid:4ba52eac-a0b4-415a-88f5-57d1fadd1aba>

bzlib/session.plt does not handle parsing cookies into uuids.

Creating a Session Object

In order to create a session object, you first need to make a dbi handle with either dbd-spgsql, dbd-jazmysql, or dbd-jsqlite driver to where you have setup the session_t table, and you need to pass in the corresponding query script so the prepared statements ('make-session!, 'load-session, 'save-session!, and destroy-session!) can be loaded:


(require (planet bzlib/session) (planet bzlib/dbi)) ;; loading 'jsqlite (require (planet bzlib/dbd-jsqlite)) (define h (connect 'jsqlite <path-to-db> '#:load (session-query-path/sqlite))) ;; loading 'spgsql (require (planet bzlib/dbd-spgsql)) (define h (connect 'spgsql <spgsql-parameters> ... '#:load (session-query-path/postgres)))
;; loading 'jazmysql
(require (planet bzlib/dbd-jazmysql)) (define h (connect 'jazmysql <jazmysql-parameters> ... '#:load (session-query-path/mysql)))

Then you can pass the handle to build-session to create the session object.


;; create a new session with a new uuid (define s (build-session h)) ;; create a new session with a known uuid (define s (build-session h <uuid>))

As soon as build-session is called, you'll find a corresponding session record in session_t.

Accessing Session Key/Value Pairs

session-ref, session-set!, session-del! modifies the values of session key/value pairs:


(session-ref <session> <key> <optional-default-value>) (session-set! <session> <key> <value>) (session-del! <session> <key>)

Writing Out Sessions

save-session! saves the sessions out to the database:


(save-session! <session>)

refresh-session! will reload the session from the database:


(refresh-session! <session>)

And destroy-session! will delete the session record from session_t:


(destroy-session! <session>)

call-with-session and with-session will help you manage the call to save-session! so you do not have to write it with every request:


(call-with-session (build-session h) (lambda (session) <... do something with session ...>)) (with-session (build-session h) (lambda () <... do something with session via (current-session) ...>))

with-session works via (current-session), which is a parameter that holds either #f or a session structure. By passing the session object via with-session it will automatically parameterize current-session.

Session Expirations

You can use session-expired? to test whether the session object has expired:


(session-expired? <session>)

The expiration value is stored as a number (in Julian days) in session_t, which by default will be 14 days in the future from the time when save-session! is called.

The default session expiration value of 14 days can be controlled via the session-expiration-interval parameter.

Web Server Continuation

To simplify the usage of bzlib/session with web-server continuation calls, the following wrappers are exported to replace the send/suspend family:


(require (planet bzlib/session/web-server)) ;; you then have access to... send/back send/finish
send/suspend
send/suspend/url
send/forward
send/suspend/dispatch
redirect/get
redirect/get/forget

These wrappers have the same corresponding APIs from web-server itself, and they'll call save-session! before making the continuation, and call refresh-session! before returning from the continuation.

That's it for now, enjoy.

Yinso Chen: Web Sessions vs. Continuations

yc (noreply@blogger.com) — Sun, 15 Nov 2009 22:29:41 -0800

The "session info in web server applications" thread recently in plt-scheme list has an undertone that continuations are equivalent of web sessions as understood in other languages and frameworks. This undertone is highlighted by the lack of a session-like capability within the web-server collection that exists in other web frameworks.

This got me to think: are continuations equivalent of sessions?

The original intent (indicated in Shriram's research paper) of web-server's continuation is to correctly and succinctly model interactive web application's application flow. The paper sites examples of incorrectly implemented web apps that would do something like the following:

user browse a list of goods
user opens new window to get the details of goods A
user goes back to original window
user then opens another new window to get the details of goods B
user then goes to goods A and click "Buy Now"
incorrectly implemented app will cause the user to buy goods B instead of goods A

The traditional solution to the above interaction would be to use sessions, and since continuation models such interactions as well, there is no question that in this case continuations supplant the needs of sessions.

But for other scenarios involving sessions it might be more natural to model the computations by using the traditional session concepts.

For example - identifying the user across visits after significant time lapse (this is generally toggled by a "remember me" checkbox during login). Normally web sites accomplish this by persisting the user's authenticators via cookies or sessions.

This process is awkward to model with continuations, since the user likely come back to the site via a top level link that has no captured continuations, instead of digging up the last continuation url for the site, and the continuations might have expired between the visits if you use stateful servlets.

If you use web-server's stateless servlet language, an approach is probably to serialize the continuation into a cookie so it can model the above scenario, but you'll have to write your code in the stateless language or convert your code over, and it feels like a more complex solution compared to simply having a regular session capability. This is similar to using continuations to model non-interactive web links - it can work, but it does not follow Occam's razor.

Furthermore - if your site uses extensive ajax, your use of continuations will decrease, since Ajax models the interactions as well and supplants the needs for continuations. and in such case you might regain the needs for sessions that was reduced by continuations.

So, as far as I can tell, continuations is not equivalent to web sessions and do not eliminate the needs for session capabilities.

Programming Praxis: Two Linear Sorts

programmingpraxis — Fri, 13 Nov 2009 01:00:47 -0800

It is well known that any comparison-based sort, as we have been studying, has a lower time bound of O(n log n). But if all the keys are positive integers less than or equal to n, it is possible to sort in O(n) linear time by taking advantage of the structure of the keys themselves.

Count sort determines, for each input element x, the number of elements less than x, then places x directly in its position in the output; if there are k elements less than x, then x belongs in the k^th + 1 position (being careful to properly consider the case of equal elements). Count sort requires two temporary arrays, one to hold the counts of the various elements, which act as indexes into the array, and one to build up the output.

Radix sort extends count sort by making multiple passes based on the positional digits of the integers being sorted: first do a count sort on the digits in the ones column of the integers, then a count sort on the digits in the tens column, then the hundreds column, the thousands column, and so on until the input is sorted, taking advantage of the fact that count sort is stable. Radix sort works on other kinds of keys besides integers; for instance, dates can be sorted by doing count sort on day, month and year in succession.

Your task is to write functions that sort arrays using count sort and radix sort. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

Dominique Boucher: The architecture of NuGram Hosted Server

Dominique Boucher (noreply@blogger.com) — Thu, 12 Nov 2009 07:12:35 -0800

In the comments section of one of my previous post, I mentioned that NuGram Hosted Server (www.grammarserver.com) is implemented in a mix of Erlang/Yaws, Java and Kawa Scheme. Alexandre Abreu then asked a few questions about some architectural aspects:

how do erlang and kawa scheme interop? http / json ? ffi ?
to follow up on that, why erlang / yaws and not scala / lift to have a better integration with the jvm ?
the server part if hosted on 1 machine ? how do you handle the load (both the connection load and the computation load since grammar and the services provided by NuGram seem pretty heavy computation-wise)? Maybe you don't need to do much given the amount of people that uses your service and I know you don't offer any guarantees with people using this service in a commercial setup but I am just wondering

These questions prompted for a long answer. So in the following sections I describe some aspects of the NuGram Hosted Server architecture at 10,000 feet high.

1. Context

NuGram Hosted Server is a grammar hosting server for use by communication applications (VoiceXML applications, IM bots, etc.). It provides various grammar-related services through a RESTful API, like dynamic grammar generation and semantic interpretation of text-based sentences (and it will soon provide robust parsing capabilities). It also provides a community-based HTML application to management and share grammars, consult application logs, etc., using standard AJAX techniques.

When we started the project, we had some ambitious requirements in mind:

High-performance HTTP server.
Scalability. (We needed the possibility of dynamically adding nodes to our cluster without interruption of service.)
Fault-tolerance.
Hot code swapping.
Distributed database.

Since I already had some experience with Erlang, I pushed the idea to the management team, after getting the buy-in from my development team. We decided to use this project to evaluate Erlang in the context of a real project.

To answer Alexandre's question more specifically, we did not consider Scala/lift mainly for the same reason we did not choose Java itself. Essentially, we could not use the usual session tracking mechanisms found in J2EE environments (jsessionid, cookies) for our RESTful API. Implementing a clustered solution would have required too much work. And using Erlang was a far better (and more elegant) approach.

2. The architecture

At a very high level, grammarserver.com is composed of two subsystems:

An Erlang/Yaws front-end.
This is the part handling the HTTP requests.
A number of Java/Kawa workers.

Let me describe each subsystem separately.

The Erlang/Yaws front-end

This subsystem handles all the HTTP requests. It currently consists of a single Erlang node, but it has been designed to support the clustering of many Erlang nodes. In this case, we would need a load-balancer in front of the Erlang cluster.

The Erlang system follows most OTP principles. It is a real Erlang application and provides a supervisor and a number of gen_servers. The application also starts an embedded yaws serving requests for several virtual hosts (the HTML websites at www.grammarserver.com, nugram.nuecho.com, with and without SSL support, and the RESTful API at www.grammarserver.com:8082).

Most requests to the RESTful API are done in the context of a session. Each session is associated with an Erlang process and the application keeps a mapping between the session ID and the process for the session in the Mnesia database (it is rather cool to store things like process IDs in a database!). So when the application receives a request, it extracts the session ID from the request URI, finds the corresponding process in the database, and simply forwards the request to that process.

When multiple Erlang nodes are running in a cluster, forwarding a request can involve sending the request to a different Erlang node. This is done completely transparently in Erlang.

Another interesting consequence of using processes to represent sessions is the fact that implementing session timeout becomes trivial. Each session process makes an explicit receive ... after. When the timeout is reached, the session is automatically terminated and removed from the database.

The application also uses Mnesia for other purposes:

It holds the user accounts, of course.
All the instantiated grammars are held in the database.
It holds the node IDs of the available Java/Kawa workers. Keeping that information in a persistent (disk-based) table of Mnesia makes it possible to shut down the Erlang application and reconnect it automatically to the Java nodes when we restart it. More on this below.

Java/Kawa workers

The Java/Kawa workers implement the basic NuGram services. They are written in a mix of Java and Kawa Scheme because most of the code is also shared with NuGram IDE, an Eclipse plugin.

The workers use jinterface to interface with Erlang. This has the advantage of exposing the workers as standard nodes to the Erlang application. In other words, the Erlang application does not even know that the workers are implemented in Java. This is completely transparent.

Many workers can be started, independently of the number of Erlang nodes. The first thing they do is try to find an Erlang node and register with it. If they cannot find an Erlang node, they wait for a specified amount of time, then try again. After a number of retries, they simply stop with an error.

Each grammar is assigned to a single worker. To distribute the load as evenly as possible across the workers, the Erlang system uses a round-robin strategy to assign workers to new grammars (if a session uses a grammar already loaded in a worker, requests are sent directly to that worker, of course).

3. Conclusion

Overall, our experience with Erlang has been excessively positive. (I have to confess that my team members already had some prior exposure to functional programming and Prolog, which helped a lot). Of course, we had to learn some things the hard way, we found some bugs in Yaws. But in the end the platform delivered on its promises. We have an architecture that can scale, we can hot swap code, dynamically change the database schema, add nodes dynamically, etc.

Since NuGram Hosted Server is a free service, we do not guarantee any quality of service, but the platform is really robust and fast and that is very important for communication applications (especially telephony applications where latency translates to dead-air during a conversation).

Programming Praxis: Merge Sort

programmingpraxis — Tue, 10 Nov 2009 01:00:56 -0800

The last O(n log n) sorting algorithm that we shall consider in our current series of exercises is merge sort. If you have two sorted sequences, they can be merged into a single sorted sequence in time linear to their combined length by running through them in order, at each step taking the smaller of the heads of the two sequences. Then mergesort works by recursively merging smaller sequences into larger ones, starting with trivially-sorted sequences of one element that are merged into two-element sequences, then merging pairs of two-element sequences into four-element sequences, and so on, until the entire sequence is sorted.

Your task is to write a function that sorts an array by the merge sort algorithm described above, according to the conventions of the prior exercise. When you are finished, you are welcome to read or run a suggested solution, or to post your own solution or discuss the exercise in the comments below.

Yinso Chen: Building a Web Session Store (1)

yc (noreply@blogger.com) — Mon, 09 Nov 2009 20:54:36 -0800

Given that we have previously determined the need for a web session store even if we are using continuations, we'll go ahead and build it on top of our DBI stack, so the session data can be persisted as long as necessary.

Quick Word About Session Store Performance

One thing to note about session data is that its data usage is both read and write intensive, and such data can put strain on the database. It's write-intensive because with each request we'll extend the expiration time on the session itself, and it's read-intensive because the data is needed for every request, but it changes with every request.

For now we'll assume that our database is capable of handling such needs (and it will until you have a sufficiently large amount of traffic), but it's something to keep in mind. The nice thing of building the session logic on top of DBI is that when we need to deal with the performance issue, we can add logics into the DBI tier easily with developing a customer driver, for example, by integrating memcached as a intermediate store that'll flush out the changes to the database once a while instead of with every request.

Active Record

The active record pattern are not just for OOP fanatics - we schemers know that you can craft your own OOP with FP easily. In DBI today there is a base structure for active record definition:


(define active-record (handle id))

Such definition is a lot simpler than the usual OOP representations, which usually try to construct the data model in memory, along with dynamically constructed SQL statements. Although such OOP records provide simplicity for the simple cases, it has proven to be a leaky abstraction due to the object vs relational paradigm mismatch, as well as a significant performance overhead. Our simple definition will do us just fine right now.

What would our session API look like then?


;; expiration a julian-day ;; store is a hash table (define-struct (session active-record) (expiration store) #:mutable) ;; the session key/value manipulation calls... (define (session-ref session key (default #f)) ...) (define (session-set! session key val) ...) (define (session-del! session key) ...) ;; the persistence calls (define (build-session handle ...) ...) (define (save-session! session) ...) (define (refresh-session! session) ...) (define (destroy-session! session) ...)

We'll go through and flesh out the definitions in details.

The Store in Memory

Hashtable is a good internal representation of the key/value pairs that session will hold (for now we'll assume the held data are serializable... we'll deal with this problem later), and this immediately tell us what session-ref, session-set!, and session-del! will look like:


(define (session-ref session key (default #f)) (hash-ref (session-store session) key default)) (define (session-set! session key val) (set-session-store! session (hash-set (session-store session) key val))) (define (session-del! session key) (set-session-store! session (hash-remove session key)))

And yes - we are using immutable hash rather than mutable hash.

When to Persist

You probably have noticed that session-set! and session-del! do not persist out to the database. So if you have multiple concurrent connections for the same session, it might be possible for the session object to get out of the sync.

While this is possible, the chance of it happening isn't great, since for the majority of the time users are going to make one main request at a time, with many auxiliary requests for accompanying images and css files that should not modify session values.

On the other hand, saving every changes with each single session-set! call could drastically increase the read & write access for the session object (what's the point of saving with each write if you are not doing the same for read?) and could have detrimental impact on performance unless we are ready to implement an intermediate cache.

And finally such decoupling actually simplify the code (I have written the code with the other approach for comparison) and makes it look more refactored. So for now we'll go with this approach.

Hence we'll persist at the end of the request with a call to save-session!. A simple wrapper so you do not have to explicitly write the separate call would be:


(define (call-with-session session proc) (dynamic-wind void (lambda () (proc session)) (lambda () (save-session! session))))

And with a current-session parameter we can simplify it as:


(define current-session (make-parameter #f)) (define (with-session session proc) (call-with-session session (lambda (session) (parameterize ((current-session session)) (proc)))))

Except for one bug, the above will work as you expected in web-server environment. If you have an idea of what the bug will be - please feel free to make a comment. I'll discuss the bug and how to fix it in the next post for the series.

Grant Rettke: Where to Find Jobs in Statistics

Grant — Mon, 09 Nov 2009 07:17:59 -0800

http://www.amstat.org/jobweb/index.cfm

http://www.stat.ufl.edu/vlib/jobs.html

http://www.stat.purdue.edu/resources/jobs/jobs.cgi?type=1

http://jobs.imstat.org/c/search_results.cfm?site_id=1847

http://eims.ams.org/search/

http://www.mathjobs.org/jobs

(via Dr. Rowe)

Yoni Rabkin Katzenell: CamelHumpSupportForEmacsMovement

yrk — Fri, 20 Nov 2009 23:59:35 -0800

I'mVeryHappyToSeeThisIncludedInEmacsForThoseTimesINeedToEditStuffLikeJavaScript.

Emacs-fu: showing pop-ups

djcb — Fri, 20 Nov 2009 09:08:28 -0800

Updated: yes, it's %s, not %d Sometimes, it's nice when emacs can warn you when something is happening or should happen. For example, when a new e-mail has arrived, or when there's a meeting in 15 minutes you should attend.

As always, there are different way to do this, but here's what I've been using for while. Various versions of this have been circulating around mailing lists, so I don't know whom to credit with the original idea – anyway, this is the (modified) version that I'm using.

(defun djcb-popup (title msg &optional icon sound) "Show a popup if we're on X, or echo it otherwise; TITLE is the title
of the message, MSG is the context. Optionally, you can provide an ICON and
a sound to be played" (interactive) (when sound (shell-command (concat "mplayer -really-quiet " sound " 2> /dev/null"))) (if (eq window-system 'x) (shell-command (concat "notify-send " (if icon (concat "-i " icon) "") " '" title "' '" msg "'")) ;; text only version  (message (concat title ": " msg))))

A couple of notes:

I'm using notify-send for sending notifications; this assumes you are using that system (it's part of the libnotify-bin package in Debian/Ubuntu). You can of course replace it with whatever is available on your system. Alternatives are zenity or kdialog or xmessage (for old-timers) and their equivalents (?) on Windows, MacOS.
I'm now using mplayer for playing sounds. This is a bit heavy, but at least plays all kinds of audio files. If you only care about .wav-files, you could replace it with e.g. aplay;
as always, please ignore my ego-centric function names :-)

Now, we can use this function by evaluation e.g.

(djcb-popup "Warning" "The end is near" "/usr/share/icons/test.png" "/usr/share/sounds/beep.ogg")

showing pop-ups from `org-mode` appointments

The above popup function is most useful when it's does its work based on some event. To be notified of appointments and the like, there is the emacs appt facility. Here, we set up this appt, and then hook it up with org-mode, so appt can warn us when there's something happening soon…

;; the appointment notification facility
(setq appt-message-warning-time 15 ;; warn 15 min in advance  appt-display-mode-line t ;; show in the modeline
 appt-display-format 'window) ;; use our func
(appt-activate 1) ;; active appt (appointment notification)
(display-time) ;; time display is required for this...
 ;; update appt each time agenda opened (add-hook 'org-finalize-agenda-hook 'org-agenda-to-appt) ;; our little façade-function for djcb-popup
 (defun djcb-appt-display (min-to-app new-time msg) (djcb-popup (format "Appointment in %s minute(s)" min-to-app) msg "/usr/share/icons/gnome/32x32/status/appointment-soon.png" "/usr/share/sounds/ubuntu/stereo/phone-incoming-call.ogg")) (setq appt-disp-window-function (function djcb-appt-display))

Of course, you can freely choose a icon / sound to your liking.

showing pop-ups for new mail

Another event you might want to be warned about is new mail. There is something to be set for not letting yourself be disturbed for new mail, but if you sufficiently filter your mails before they enter your inbox, it can be a good way to periodically bring you back from your deep sl ^H^H thinking. For Wanderlust, I use something like this:

(add-hook 'wl-biff-notify-hook (lambda() (djcb-popup "Wanderlust" "You have new mail!" "/usr/share/icons/gnome/32x32/status/mail-unread.png" "/usr/share/sounds/ubuntu/stereo/phone-incoming-call.ogg")))

Exercise for the reader: adapt this for your chosen mail client.

Got Emacs?: Churn baby, Churn

sivaram — Fri, 20 Nov 2009 07:52:40 -0800

Rs 19.

That's the fee for number portability. Less than a half pack of ciggies cost. With a purported 400 million user base, it's going to be one massive headache for the operator. Much of the problem is one of their own making. Increasing the subscribers without attendant increase in cell phone towers. Dropped calls and no connectivity, with people on prepaid watching their precious talk time go down without a word being spoken, they're going to get socked like crazy by the prepaid crowd.

Everyone of my friends and colleagues are just waiting for this to get rolled out and then, they'd hit the phones and switch. I believe the salesmen who work on commission to close out on subscribers for each operator will make a killing in the first few months.

Though, I'm pretty sure, the operators will find a way to make it a disincentive to switch. One rumour I heard was that it might take as long as 4 or 6 weeks for the switch to be complete. If the US experience is considered when portability was introduced, I'd wait a good 6-8 months before switching.

If it comes to that. I'll take a lower bill and other extra minutes thrown in, anytime rather than switch. Since I have my broadband and mobile from the same operator, hopefully I have some leverage.

Hopefully.

Random Sample: Posting org buffers with weblogger.el to your blog

randomsample — Fri, 20 Nov 2009 07:08:04 -0800

In my last post, I described how to setup weblogger.el with Drupal. Since I love to write posts like this with org (see this posting for details on how you can do this), I wrote myself a little helper function to directly export an org file to a weblogger entry:

(defun DE-org-export-weblogger () (interactive) (let ((tmpbuffer (get-buffer-create " *org html export*")) title text) ;; export posting to HTML, but without headers
 (org-export-as-html 1 nil nil tmpbuffer t) (set-buffer tmpbuffer) (goto-char (point-min)) ;; get the title
 (when (re-search-forward "<div id=\"outline-container-1\" class=\"outline-2\">[^\0]*\
<h2 id=\"sec-1\">\\(.*?\\)</h2>[^\0]*\
<div class=\"outline-text-2\" id=\"text-1\">" nil t) (setq title (match-string 1)) (replace-match "")) ;; get the posting
 (setq text (buffer-substring-no-properties (point) (point-max))) (weblogger-start-entry) (insert title) (goto-char (point-max)) (insert text) (kill-buffer tmpbuffer)))

You can now simply call DE-org-export-weblogger interactively on your org file, and it will create a new weblogger entry buffer. Determining the title of the posting is a bit hacky; it assumes that your org file looks something like this:

* Title of your blog post text of your blog post

It will then extract the title from the HTML exported output. You can still edit the produced HTML, maybe inserting your taxonomy tags, and then post it by pressing C-x C-s.

This of course assumes that your default input format for blogs is "Full HTML", otherwise most of the HTML tags will get stripped and everything will just look plain ugly. You can set the default input format in Drupal under site configuration -> Input formats.

BTW, I totally forgot to mention in my last post that you can also edit existing Drupal blog entries with weblogger. Simply press C-c C-p or C-c C-n for the previous/next entry!

Chris Ball: Btrfs snapshots proposal

Chris Ball — Thu, 19 Nov 2009 10:40:00 -0800

I've written up a feature proposal on how we can use Btrfs snapshots to enable system rollbacks in Fedora 13, by gluing together the existing kernel code to do Btrfs snapshots, a UI for performing rollbacks, and a yum plugin to make snapshots automatically before each yum transaction. Lots of good comments so far, and LWN has written an article about it (subscribers only, for the moment).

Random Sample: Posting Drupal blog entries from within Emacs

randomsample — Thu, 19 Nov 2009 07:07:38 -0800

I just found out you can easily post Drupal blog entries from within Emacs by using the wonderful 'weblogger.el'. Here's what you have to do:

Enable the 'Blog API' module in your Drupal installation and configure permissions and the blog types which should be accessible via XML-RPC.
Go to the Weblogger EmacsWiki page and get yourself weblogger.el and xml-rpc.el.
Install both in your load-path and require weblogger in your .emacs.
Now call weblogger-setup-weblog and use https://your-drupal-site/xmlrpc.php as URL for accessing the Drupal XML-RPC API, then provide username and password.
Now call weblogger-start-entry and start writing.
Use C-x C-s to post.

That's it! Now you can for instance create blog entries with the fantastic markup from Orgmode, and directly post it via weblogger-mode. There's just one problem: how can you set the taxonomy tags for the posting? Simply putting it into the 'keywords' section didn't work. It seems like you cannot do that via XML-RPC in Drupal, but luckily there's a little Drupal module called 'inline tags', which lets you flag the posting through a special tag

[tags]Emacs[/tags]

Note that the module requires Drupal 6, but it wasn't very hard to make it work under Drupal 5 (which I'm still using) - let me know if you need it.

Matt Harrison: New insight on 100% coverage

Matt — Wed, 18 Nov 2009 08:56:00 -0800

Code coverage gets a bad rap. People say it's meaningless. In my "Managing Complexity" talk at PyCon 2008, I made the analogy that code coverage was like a shield. One is exposed where the shield doesn't cover. But just because you have shield doesn't mean that your shield is any good. You could be battling against a fire breathing dragon with a WOODEN shield (smart one Wiglaf). Your exposed areas could be dead code (which the literature says is more likely to cause bugs) or code that is hard to test.

This last point, code that is hard to test was hit home to me over the weekend when I spent some time going over Miško Hevery's blog about testability. While perusing the site and some of his presos, it hit me that 100% code coverage (and I'm talking about the line/statement variety here) indicates something else. That your code is testable. If you can't get 100% coverage, then you are probably violating some of the 10 things that make code hard to test. I especially appreciate the Law of Demeter analogy he uses, (paraphrasing) "You don't give the cashier your wallet when they tell your lunch costs $5.98. You give them the money (or the credit card)". So if you are struggling getting the coverage numbers you want, you probably need to use dependency injection to get rid of that Law of Demeter violation....

Of course, now that Python has branch coverage. we can get to work on some of the interesting things that you get from path coverage. Oh yeah, editor support would be nice too.

Bryan Murdock: Initial Thoughts on Google Go

Bryan — Wed, 18 Nov 2009 08:29:58 -0800

I read the “tutorial”: and thought, “the syntax is Frankenstein ugly. They have Guido on staff, but this looks like they consulted Larry Wall. Why didn’t they just write an open source D compiler?”

Then I read the FAQ. Some very interesting ideas, and personally, ever since using C++ for some large embedded programming projects (large? embedded? not as oxymoronic as you might think), I’ve been thinking these same things for a while now:

No major systems language has emerged in over a decade, but over that time the computing landscape has changed tremendously. There are several trends:

Computers are enormously quicker but software development is not faster.

Dependency management is a big part of software development today but the “header files” of languages in the C tradition are antithetical to clean dependency analysis—and fast compilation.

There is a growing rebellion against cumbersome type systems like those of Java and C++, pushing people towards dynamically typed languages such as Python and JavaScript.

Some fundamental concepts such as garbage collection and parallel computation are not well supported by popular systems languages.

The emergence of multicore computers has generated worry and confusion.

We believe it’s worth trying again with a new language, a concurrent, garbage-collected language with fast compilation.

D has been around a while, but it just doesn’t seem to be catching on; maybe because it just doesn’t have a large corporation behind it. Maybe because it’s not fully open source. Or, maybe because it doesn’t have all the features of Go.

Bryan

Yoni Rabkin Katzenell: There is Very Little Good on the Web; Keep it Alive

yrk — Wed, 18 Nov 2009 08:06:06 -0800

I don't think that Wikipedia is particularly accurate or run by saints but it is an important part of what makes the Web useful. So I donated 10 bucks as a part of their new fund drive and I think that you should too.

sachachua: Automating tedious wiki editing tasks with Emacs and w3m

Sacha Chua — Tue, 17 Nov 2009 11:24:31 -0800

I needed to update many of the links in our wiki because a team member left, so I had to reupload all of her files to a shared service and change all the URLs to point to the new files. Unfortunately, the file service didn’t send me the former URLs of the files, so that was going to be a manual process. Our wiki had 149 pages in it. Not fun.

After a few pages of editing (and correcting the occasional typo that crept in as I changed URLs), I decided to partially automate the process. Using a smidgen of Emacs Lisp, I created a function that pasted text into a temporary buffer, performed whatever automatic fixes it could make, prompted me for any URLs it didn’t recognize, remembered the old URL – new URL mapping I defined, and copied the text back.

The function looked somewhat like this:

(defvar sacha/wiki-links nil "Associative list of (old-url . new-url).")
(defun sacha/wiki-fix () (interactive) (with-temp-buffer ;; Insert text from clipboard
 (yank) (goto-char (point-min)) ;; Look for all the links
 (while (re-search-forward "\\[\\([^|]+\\)|\\([^\]]+\\)\\]" nil t) ;; Check if it's one of the links I want to replace
 (if (or (string-match-p "viewpage" (match-string 2)) (string-match-p "lsoohoo" (match-string 2))) (replace-match (save-match-data ;; Prompt and the entry to the map if it does not yet exist
 (unless (assoc (match-string 2) sacha/wiki-links) (add-to-list 'sacha/wiki-links (cons (match-string 2) (read-string (concat (match-string 1) "? "))))) ;; pick up the corresponding URL
 (cdr (assoc (match-string 2) sacha/wiki-links))) t t nil 2))) ;; Copy the text into the clipboard
 (kill-new (buffer-string))))

I used M-x global-set-key to bind a convenient function key to it (F12, I think), and then it was just a matter of clicking on each page, clicking on Edit, typing Ctrl-C to copy the text, switching to Emacs, pressing F12, switching back to my browser, typing Ctrl-V, and saving the wiki page. I also added some lines (not shown here) to convert the previous wiki gardener's full links to intrawiki links, change server URLs, and do other fun things.

I thought about fully automating it (somehow hooking into w3, perhaps?), but that seemed to be more trouble than needed. Besides, it was good to review all the pages.

As a result of this Emacs wizardry, processing all 149 wiki pages took me a few hours instead of a few days. Yay!

Of course, I finished the last wiki page, I found out that I needed to change the servers in the URL. I decided to go ahead and fully automate the darn thing.

I extracted a list of URLs for the wiki by viewing the tree version of the wiki index. It used Javascript, so I couldn't just pull the URLs out of the source code. Fortunately, the Firebug plugin for Firefox lets me copy the rendered HTML, so I used that instead. Some judicious text-editing later (replace-regexp rocks), I had a list of URLs to the different pages. I knew I needed to put in some kind of delay when loading web pages. sleep-for let me spread out my requests so I didn't hammer the server too badly. Reading the w3m.el source code turned up w3m-async-exec. Once I set that to nil, requesting web pages and running code on the results turned out to be straightforward. Selecting the right widgets was a bit of a hack (re-search-forward here, w3m-previous-anchor there), but hey, it worked. After confirming it by manually running it on a few pages, I left it merrily running in the background.

Here it is (some tweaking required):

(defun sacha/edit-wiki-page () (interactive) (let ((buffer (current-buffer)) (w3m-async-exec nil) (delay 5)) ;; number of seconds
 ;; While not at the end of the buffer
 (while (not (eobp)) ;; Load the URL on the current line
 (w3m-browse-url (buffer-substring (line-beginning-position) (line-end-position))) ;; Look for the edit button
 (goto-char (point-min)) (when (search-forward "Edit" nil t) ;; Click it
 (w3m-view-this-url) ;; Look for the Minor change checkbox
 (goto-char (point-min)) (when (search-forward "Minor change" nil t) ;; The text area is the second widget back
 (w3m-previous-anchor 2) ;; Open the text area in a temporary buffer for editing
 (w3m-view-this-url) ;; Do the changes
 (while (re-search-forward "https?://example.com/path" nil t) (replace-match "http://path.example.com" t t nil 0)) ;; Save the value
 (w3m-form-input-textarea-set) (when (search-backward "Save" nil t) (w3m-view-this-url)))) (switch-to-buffer buffer) (forward-line) (sleep-for delay))))

I’m sure this kind of automation might be possible with lots of hacking in Mozilla Firefox, and I’ve seen great scripts for the Mac, too. But I know Emacs, I’m comfortable digging into source code, and I can make things work.

Awesome. =D

Post from: sacha chua :: enterprise 2.0 consultant, storyteller, geek. Check out my blog for tips on managing virtual assistants, Drupal, and other topics!

Automating tedious wiki editing tasks with Emacs and w3m

Alex Bennée: Thinking about email

Alex — Tue, 17 Nov 2009 04:49:55 -0800

I saw a post on my feeds today discussing a new email client called notmuch which set me thinking about my own use of email.

I have three primary ways of accessing email at the moment. On my domain I run Mutt in a screen session. At work I have the GUI behemoth that is Evolution and most of my mailing list subscriptions run through a Gmail account I have set aside for the very purpose.

I’m quite happy with Mutt for my main personal email. As I don’t use my personal email to subscribe to mailing lists the volume is low, it’s fairly quick and snappy to write email (in an emacsclient naturally) and searching is only a “/” away. However it is intrinsically a folder based email client which works fine as I only have two folders: Inbox and Oldmail. I periodically (~annually) move email into Oldmail when Mutt starts to slow down on start-up.

The main bugbear is my work email. I mainly run evolution through inertia and 2 features:

It connects to the work Exchange server, although calendering doesn’t really work properly as I can’t Accept/Decline meeting invites.
The vFolder concept allows me to easily separate company emails from the bug tracker and CVS commits and the odd work related mailing list while still offering a quick eyeball indication of what pending emails there are.

Other than those features Evolution sucks balls in every other way. It takes an age to compose a new email, I have to shut it down every night to avoid swap death and it’s certainly not keyboard operable.

The GMail account I have to live with as there isn’t really anywhere else I can easily index and search the 4gb (and growing) of mailing lists I subscribe to. Of course it is accessible via IMAP so it would be nice if I could access it from the same client, especially as the web interface tends to munge in-line patches.

The obvious choice would be to look into one of the many many modes for handling email in emacs. However the plethora of choices is one that makes selecting one to try even harder. Some people at work have fetchmail tasks setup to mirror the email on the server which sounds a little clunky but I guess must work well enough. I don’t think it would be practical for Gmail though, I usually just browse through various tags for mailing lists, including the very handy ‘mythreads’ which marks any email list thread I’m involved in.

I think the key feature I need will be something like Evolutions vFolders for the quick eyeball along with a decent way to interact with Gmail. Any suggestions?

Joseph Miklojcik: Lexical Analysis

jfm3 — Sun, 15 Nov 2009 07:41:39 -0800

Lexical analysis turns a stream of input characters into identifiable chunks; "lexemes". For example the stream of input "3" should turn into the lexeme for the number three. Decisions about the syntax of a language must be made at the lexical level to disambiguate stream of input such as 10E-6. One language may intend for such a stream to mean a single number, namely ten times 10 to the power of negative six. Another may instead intend two numbers and a symbol; ten, E and negative six.

The main decision in designing a lexical system is usually about how to divide lexemes into names, numeric literals, special punctuation, and comments. If the characters that can go in each of these categories do not overlap, separation between lexical elements can be more free in programs. If there are few restrictions on what goes in a name or numeric literal, then disambiguating them from their surroundings requires more care.

For example, imagine language A has the lexical rules; "names may consist only of letters" and "numbers may contain only decimal digits". More complicated numeric literals presumably are constructed using other language features. With these two lexical rules, we can give special lexical significance to single characters that can represent common arithmetic operations. a3 generates two lexemes, a and 3. The program fragment a+3 is read as three lexemes. Programmers are free to use white space as they choose. Transitions between lexemes are accounted for entirely by the contents of the lexemes themselves, unless two numbers or two names occur in sequence.

Now imagine a different language B, which has the lexical rules; "names may consist of any characters other than white space" and "numbers are names which contain only decimal digits". Again, this glosses over more complex numeric literals. Under language B's lexical rules, the program fragment a+3 is a single lexeme (as is a3). To generate three lexemes, the programmer must explicitly separate the lexical elements using white space; a + 3. However, in this case + need have no special significance. It can be a name like any other.

A third general approach to lexical analysis could be characterized by a language C, which has more complicated lexical rules that describe when lexemes begin and when they end. Language C's rules are: a name begins with something that is not a digit, and ends when a space or special character is encountered. A number begins with something that is not a letter, and ends when a space or a special character is encountered. Such a lexical system interprets a+3 as three lexemes, but a3 as one.

Languages A, B, and C as presented above are greatly simplified and gloss over a large number of important issues. But judging these rudimentary forms, it would seem that languages A and C have the preferable quality of doing the right thing if a programmer omits spaces, language C more so than language A. However, the more important question is, which of language A or B conforms more to how the human mind processes language? In my opinion, language B comes closer.

In terms of orthography, we know that words are things human readers take in as units separated by spaces. The well known trick of changing the order of letters in words save for the first and last letter, and observing that such words are still somehow comprehensible, demonstrates this;

"Tihs is an eaxpmle of scuh a senetcne."

Compare your ability to comprehend the above sentence to the one in which letters are all in the correct order, but spaces are omitted;

"Thisisanexampleofsuchasentence."

The eye of a reader can go faster (and correct more errors) if lexemes that produce a sound when read aloud are separated by space. Demanding such separation in a programming language therefore makes it conform to the human eye. Most programming languages which allow a great deal of freedom in the omission of spaces between lexemes also have earnest style guides which encourage programmers to use spaces to separate the lexemes anyway.

In terms of natural language, we chunk lexical elements from either language A or B into several utterable chunks, as in "ey plus three", or "ten times ten to the power of negative six". The amount of compression into the shorthand 10E6 is similar in A, B, and C, whether we demand that it be written 10 E 6 or not. Lexical issues in programming languages must focus more on making an unambiguous orthography than on, say, morphological rules in natural languages.

The exception to the natural use of spaces to separate lexemes comes when we want to form compound names. Many natural languages allow putting multiple nouns together into a single word both in spoken and written form. Sometimes the programmer wants to name something with a single programming language lexeme that contains multiple names, for example, the fifth element of an array named R might be R[5]. It seems unnatural to write R [5], since we are naming one thing, just with compound terms.

Rosebud defines several characters which can not go in names, including white space characters. Lexemes are separated by these characters. These include characters that are used to form compound names, and characters that indicate program structure. Except for white space, these single characters are all themselves lexemes. By convention, as in written English, a hyphen is used to construct names that are compound in meaning to the programmer, but singular in meaning to the author.

Whether UTF-8 or UTF-16 would be a more compact encoding for Rosebud programs is impossible to say. UTF-8 eliminates byte ordering issues. So, Rosebud interprets its input as a stream of bytes in UTF-8. Rosebud has enough syntactic extension facilities so that if large amounts of data that would require three bytes in UTF-8 but only two in UTF-16 needed to live in program files, a UTF-16 encoding could be used for it.

Joseph Miklojcik: jJava: a possible Phosphoros competitor?

jfm3 — Sun, 15 Nov 2009 07:32:42 -0800

The Phosphoros team is uncertain as to whether or not jJava is a serious competitor to our own offering. A well known member of the Lisp community and the Java specification team is sometime quoted as writing about Java:

we were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp.

we fear that developments in the area of jJava will drag a lot of innocent programmers halfway back to C++.

We are also curious to see how they will solve the Gosling Tarpit problem.

Yann Hodique: https traffic logger

Yann Hodique — Sun, 15 Nov 2009 03:21:51 -0800

I'm currently writing a Simulator (really a mock) for some server, and more precisely for the SOAP interface it provides. All traffic is done only in https, and for debug purpose I needed to dump traces of the communication between a client and the real WebService, to compare them with the communication I have between the same client and the fake WebService. So here is a tiny script that acts as a proxy to unwrap/log/re-wrap SSL stuff.

http://gist.github.com/224171

I have the feeling all this should be doable by using ncat only, but did not find how yet.

Read or add comments to this article

Yann Hodique: s5 presentation from org-mode

Yann Hodique — Sat, 14 Nov 2009 12:21:33 -0800

Generating S5 based presentations from an Org Mode file is something I've considered for a while. I've just published some working code to do so: http://github.com/sigma/org-s5 The README file should give the basic steps to make it work.

Constraints I wanted to respect:

no modification needed on Org-mode or S5
easy to reuse the glue across multiple presentations
adaptable enough to survive multiple S5 themes

This is probably far from being complete, but at least it's usable enough for me to do presentations at work.

Hope it can be useful for someone else.

emacs

Read or add comments to this article

Thomas Kappler: Announcing md-readme

thomas11 — Sat, 14 Nov 2009 08:56:22 -0800

You can find quite a few Emacs libraries on github these days. I only have one very small project there, wpmail, which I haven’t written about yet because I want to finish the Markdown support first… anyway: something I find really neat on github is the automatic display of the README file in the repository’s root directory, if it exists. Only, as we all know, writing READMEs is tedious and we’d rather spend our time programming. But wait: in the Emacs world, we’re actually writing a README for each project: the library headers. Their content and structure are rather clearly defined, and it’s expected for all Emacs libraries. So I had the idea of generating a nice, Markdown-formatted README.md for github from these headers.

I just pushed the first version of md-readme that does just that. The README that you see on the github page is auto-generated from md-readme.el, with lists, code-block and everything.

The conversion to Markdown is of course pretty simple, but even so it’s pretty cool to see how little code I had to write. Emacs provides everything out of the box. (with-temp-file "README.md" (do-it)) is just elegant.

Now, seeing that we have code that generates a nice README from our Lisp file, it’s natural to have it run automatically so we always have an up-to-date README. Here I was struggling for a while. I had lots of ideas, but none seemed to work out. My current solution is setting a per-directory local variable for each of your ELisp projects that triggers the conversion via an after-save-hook. Code is in the README. The problem with adding the after-save-hook to all modes is that it’s evaluated all the time, for each save. Kinda annoying, even if it’s just a single condition.

Does anyone have a better idea? Do it outside Emacs in a git pre-commit hook?

But even with the current solution it works, and that’s the important part. Please try it, and enjoy your new READMEs.

Posted in Own Code Tagged: elisp, emacs, github, markdown, Own Code, readme

Justin Heyes-Jones: Transparent emacs on windows

Justin — Fri, 13 Nov 2009 22:30:24 -0800

Here's a handy function which lets you choose a transparent level (0 is fully transparent and 100 is opaque), for the main emacs frame in both focused and unfocused state.

(defun transparent(alpha-level no-focus-alpha-level)
 "Let's you make the window transparent"
 (interactive "nAlpha level (0-100): \nnNo focus alpha level (0-100): ")
 (set-frame-parameter (selected-frame) 'alpha (list alpha-level no-focus-alpha-level))
 (add-to-list 'default-frame-alist `(alpha ,alpha-level)))

To run M-x transparent.

Enter the values you want for when the window has focus and when it does not. In the background you can see system status information on the desktop, which is another excellent utility from sysinternals called BgInfo. It let's you display useful info about your system right on your desktop, similar to tools you may find on linux.

Phil Hagelberg: in which things are mapped, but also reduced

Phil Hagelberg — Fri, 13 Nov 2009 12:29:30 -0800

A while ago Tim Bray had a project called the Wide Finder to collect implementations of a log parsing problem in different languages to see which ones made it easy to take advantage of massively-parallel hardware. The idea is to accept a web server log file and return statistics for which pages have been requested the most. Yesterday he posted a follow-up in Clojure using refs.

While his version is interesting for someone just getting into the language because it uses refs (probably the sexiest piece of the language), I think a map/reduce approach is a little more idiomatic since it can be done with no explicit state change.

I'll step through my implementation piece-by-piece. Note that it is naïve and could be optimized for speed at the expense of straightfowardness, especially with regard to reading from disk; my intent here is simply to explain the functional approach.

Update: Commenters have posted a version that totally smokes my agent-based implementation in terms of performance and simplicity. I'm leaving it up because I think it's a fun romp in the land of higher-order functions; if you think having higher-order functions means "I can pass a block to Array#map in Ruby" then you're in for a treat.

(ns wide-finder "A basic map/reduce approach to the wide finder using agents. Optimized for being idiomatic and readable rather than speed." (:use [clojure.contrib.duck-streams :only [reader]]))

Every Clojure file opens with a call to the ns macro. This defines a namespace (wide-finder) and states any dependencies it may have on other namespaces, in this case the reader function from the duck-streams contrib library. Omitting the :only clause would cause it to make all the duck-streams vars available in this namespace, but it's usually better to be explicit about what you need.

We'll start with the entry point. The find-widely function below takes a filename and a number of agents to work with. Agents can be thought of as independent asynchronous workers that share a thread pool and keep a single state value. They are initialized with the agent function that takes a starting state. We'll be using each agent to keep a map of pages to hit counts, so we map this function over a list of n empty maps generated by repeat to get our list of initialized agents.

(defn find-widely [filename n] (let [agents (map agent (repeat n {}))] (dorun (map #(send %1 count-line %2) (cycle agents) (line-seq (reader filename)))) (doseq [a agents] (await a)) (apply merge-with + (map deref agents))))

Once the agents are initialized, we send them work. Our line-seq sets up a lazy sequence of lines from the file. We map over this sequence together with an infinite loop of the agents we construct using cycle. The map function can loop over multiple sequences in parallel and will stop when the shorter one runs out, so this is just a way of pairing each line in the file with an agent in a round-robin fashion.

(dorun (map #(send %1 count-line %2) (cycle agents) (line-seq (reader filename))))

The code that does the mapping is the anonymous function #(send %1 count-line %2), which adds a call to the count-line function to the agent's internal queue. This could be written in more traditional lambda form as (fn [a line] (send a count-line line); the #() form is simply shorthand. The count-line function will be called with two arguments: the agent's current value and the next line in the seq. That function will return a new value for the agent.

(doseq [a agents] (await a))

Once all the work has been queued up, it's necessary to call await on each agent to make sure it has a chance to finish its queue. If we used map here, it would be a no-op, since map merely creates a lazy sequence; it does not actually perform the function calls until the value is needed. Since it's not in the tail-call position, the value would simply be discarded. Note that the same problem would occur in the previous call to map, but wrapping it in a dorun call forces the lazy seq to be evaluated.

(apply merge-with + (map deref agents))

Once that's done we can call deref on each agent to get its value. But this gives us n maps rather than a list of totals, so we need to merge the values. merge-with is a special case of reduce that assumes it works on a sequence of maps and merges key collisions using the provided function, in this case +. This gives us a map of page names to hit counts.

(defn count-line [counts line] (if-let [[_ hit] (re-find #"GET /(\d+) " line)] (update-in counts [hit] inc-or-init) counts))

Finally we have the function that actually performs the counting. As mentioned, it takes an agent's state (which is the current map of pages to hits) and a line from the log file. If the line matches the regex #"GET /(\d+) ", then we return an updated version of the counts map that increments the entry corresponding to the hit. The other interesting thing here is that if-let uses destructuring: by binding the return value of re-find to a vector form, it splits the value in two. The first element (the full matched string) is bound to the unused _ local, and the second element (the match group corresponding to the actual page path) is bound to hit.

(defn inc-or-init [i] (if i (inc i) 1))

The only piece left is the tiny inc-or-init function that increments the counter given it but treats nil as zero. This would be unnecessary if we could construct hash-maps with custom default values, which a perusal of the implementation of PersistentHashMap.java seems to indicate is supported, though it's not exposed anywhere through a Clojure function. This is the piece of the puzzle that I'm the least happy with, but it may be possible to eliminate it.

In any case, this is a simple example of how to break up a commonplace problem using a classic map/reduce strategy and immutable data structures. I have no idea how it compares in terms of performance to the other Wide Finder implementations since the logs I have for this site are not exactly the hundred-megabyte blockbuster kind of logs that ongoing enjoys, but the fact that it can be parallelized in twelve lines is a testament to the expressiveness of the language.

Emacs-fu: copying lines without selecting them

djcb — Thu, 12 Nov 2009 11:27:32 -0800

When I'm programming, I often need to copy a line. Normally, this requires me to first select ('mark') the line I want to copy. That does not seem like a big deal, but when I'm in the 'flow' I want to avoid any little obstacle that can slow me down.

So, how can I copy the current line without selection? I found a nice trick by MacChan on EmacsWiki to accomplish this. It also adds ta function to kill (cut) the current line (similar to kill-line (C-k), but kills the whole line, not just from point (cursor) to the end.

The code below simply embellishes the normal functions with the functionality 'if nothing is selected, assume we mean the current line'. The key bindings stay the same (M-w, C-w).

To enable this, put the following in your .emacs:

(defadvice kill-ring-save (before slick-copy activate compile) "When called interactively with no active region, copy a single line instead." (interactive (if mark-active (list (region-beginning) (region-end)) (message "Copied line") (list (line-beginning-position) (line-beginning-position 2))))) (defadvice kill-region (before slick-cut activate compile) "When called interactively with no active region, kill a single line instead." (interactive (if mark-active (list (region-beginning) (region-end)) (list (line-beginning-position) (line-beginning-position 2)))))

It also shows the power of Emacs-Lisp with the defadvice-macro – see the fine documentation. Using defadvice, you can 'decorate' any function with your own modifications. This great power should be used with caution, of course, as to not break other usage that assumes the undecorated versions. In this case, that seem unlikely. And note that the 'advise' only applies when the functions are called interactively.

Flickr tag 'emacs': Tidy up clojure code in TextMate

manjilab — Thu, 12 Nov 2009 09:31:27 -0800

manjilab posted a photo:

Indentation of Textmate (for Lisp/Clojure) is poor.

This is how to tidy up your code with emacs-style indentation.

Flickr tag 'emacs': Tidy up clojure code in TextMate

manjilab — Thu, 12 Nov 2009 09:19:14 -0800

manjilab posted a photo:

Indentation of Textmate (for Lisp/Clojure) is poor.

This is how to tidy up your code with emacs-style indentation.

Bryan Murdock: I Got a Mac

Bryan — Wed, 11 Nov 2009 11:58:11 -0800

My employer just supplied me with a MacBook Pro running OS X Version 10.6.1 (is that a little redundant?). For a long time I have eyed macs with wary suspicion, but with a lot of curiosity too. If Microsoft is evil, as some like to say, because it locks customers in to its proprietary software, Apple must be at least twice as evil. They lock customers in to their software *and* hardware! I kind of like the new laptop, though. Mechanically, it's awesome. The aluminum body feels nice and solid. The lid doesn't latch when it closes, it just kind of (magnetically?) holds to the body. The buttons and USB slots and everything are also very nicely done. The display is one of those glossy ones for "brighter colors" and I hate the glare. The user interface has me all confused with buttons and menus in places I'm not used to. It has this fancy new mouse trackpad that is huge and it lest you do multi-finger stuff kind of like an iPhone (which I don't have). That's all going to take some getting used to also.

The biggest thing that surprised was the childhood memories that it evoked. The first computer I ever used was a Mac. Grandpa is a chemical engineer, and he recognized right away how useful personal computers would be. He had a Mac Classic in his den and he would let me play with it all I wanted when I came to visit. MacPaint was about all I could do with it when I was 5 years old, but I moved the mouse and clicked on things, pushed a floppy disk into its drive and then drug the icon to the trash to eject it, and poked around at whatever I could. On this new laptop when I first saw the little mouse pointer turn to the text entry icon, the same icon that the Mac classic used, all the memories of that Mac Classic in Grandpa's den came back. There are a few other subtle little things, like the apple menu, that give me a little bit of nostalgic joy every time I encounter them as well. It's very cool that Apple has kept some things the same over all those years. Apple can't be totally evil, right?

Got Emacs?: Another Day, Another Language: Go: From Google

sivaram — Wed, 11 Nov 2009 09:19:24 -0800

From LWN, I learnt that Google has released a new systems language because....nothing had been done in that area for some time.

So, we have Go.

After you've gone through the FAQs, which I strongly suggest you do, as it contains interesting pointers on the whys and whats of this language, one has this nagging feeling ....

Exactly what issue are they addressing and for whom? It's there in the FAQ but are YOU convinced?

Phil Jackson: A Magit clone

Phil Jackson — Sun, 08 Nov 2009 10:55:46 -0800

I’ve got a clone of Magit which I’ve been hacking away at. It’s mostly a bunch of git-svn fixes/features as that’s what I use for work but there’s some other stuff that might be of interest. It includes the following branches (all merged into their master at gitorious):

fancier-diffs: Some basic highlighting of the commit sha and of the diff markers (just the diff markers, not the whole line).

fancy-log-pt1: Some new faces for the log buffer: magit-log-graph, magit-log-sha1, magit-log-message, magit-log-head-label-remote, magit-log-head-label-tags and magit-log-head-label-local. They should give a clue as to what this branch is for…

fancy-log-pt1: Basically makes git-svn work again.

squashable-merges: Giving a prefix arg to magit-merge will allow you to squash it (handy for git-svn if your colleagues aren’t used to seeing lots of commits in trunk).

magit-completing-read: Lets you use anything-completing-read or ido-completing-read for… completion.

svn-find-rev: Hitting N f with prompt for an SVN revision and give you the commit if it’s in the current branch. With a prefix arg you will be prompted for a branch in which to look.

Like I said, all of the above is in master so you can just download magit.el from there and you’ll get it all.

Matt Keller: Switching to Atom from RSS

Matt Keller — Sun, 08 Nov 2009 03:43:19 -0800

I've added an Atom feed for this blog that carries complete xHTML content. The old RSS feed carries just simple text. I'll leave the RSS feed in place for the time being, but I will likely pull the plug on it in the future.

I've also created an Atom feed that carries just my Emacs-related content. This might be useful for anyone following mk-project happenings.

Both feeds should be auto-discoverable by your browser.

--Matt

Thomas Kappler: Now focus!

thomas11 — Fri, 06 Nov 2009 13:45:06 -0800

I’m having fun with this bit of Emacs Lisp that turns your current buffer in a fullscreen, minimal, no-frills frame a la Writeroom. I got it from the EmacsWiki and made a few small modifications.

Now, whenever you feel distracted, tell your Emacs (and thus yourself, as Emacs is a direct extension of your brain): now-focus!.


(defun now-focus! () "Open the current buffer in a frame without any bling." (interactive) ;; to restore: ;; (setq mode-line-format (default-value 'mode-line-format)) (let ((frame (nowfocus-make-minimal-frame))) (select-frame frame) (setq mode-line-format nil) ;; for Windows, untested (when (fboundp 'w32-send-sys-command) (w32-send-sys-command 61488 frame)))) (defun nowfocus-make-minimal-frame () (make-frame '((minibuffer . nil) (vertical-scroll-bars . nil) (left-fringe . 0) (right-fringe . 0) (border-width . 0) (internal-border-width . 64) ; whitespace! (cursor-type . box) (menu-bar-lines . 0) (tool-bar-lines . 0) (fullscreen . fullboth) (unsplittable . t))))

If you’d like some more options, turn to darkroom-mode.

Posted in Stuff Tagged: elisp, emacs, programming, Stuff

@emacs: emacs: Happy 3 years of Planet Emacsen! http://planet.emacsen.org/

@emacs — Wed, 04 Nov 2009 13:27:58 -0800

emacs: Happy 3 years of Planet Emacsen! http://planet.emacsen.org/

Edward O'Connor:

Edward O’Connor — Wed, 04 Nov 2009 08:50:27 -0800

I launched Planet Emacsen three years (and five days) ago. Since then I’ve enjoyed reading all sorts of interesting things about how other people live in Emacs, and I hope you have too!

If you’d like me to add your Emacs feed to Planet Emacsen, send me an email and I’ll get right on it.

Nathaniel Flath: Java and C++ Utilities

nflath — Wed, 04 Nov 2009 07:42:42 -0800

I’ve been working on some utilities for coding in Java. JDE and CEDET ground my emacs to a halt the last time I tried them, so I wanted something lightweight. So far, I mostly have some functions for looking up documentation - including c++ documentation - that I store locally on my computer and keep in a repository, but I also have a few utilities for auto-importing Java classes.

The utilities to follow need these macros defined. I talked about them previously at:
http://nflath.com/2009/08/emacs-timing-and-upgrades/. They are utilities for generating functions that take arguments defaulting to word at point.

(defun my-fn (fn prompt) "When given a function taking one argument and applying a function to it, will use that function and default to the word at point, with a prompt including that word." (let ((default (current-word))) (let ((needle (read-string (concat prompt " <" default ">: ")))) (if (equal needle "") (funcall fn default) (funcall fn needle))))) (defmacro defun-my (name prompt &rest body) "Will define both a function and a my- version of the function,
which defaults to the word at point." `(progn (defun ,name (arg) ,@body) (defun ,(intern (concat "my-" (symbol-name name))) () (interactive) (my-fn (quote ,name) ,prompt))))

These functions will be used in some of the later functions I wrote. These are used for caching large directory structures in a buffer to search for files instead of parsing the output of ‘find’ each time. Specifically, I use these to quickly look up which file I should be referencing to view documentation on Java and C++ classes. create-file-list will just create a list of files in the given buffer, and find-location-for-doc-from-buffer will return the full path of the matching html file you are searching for. Java-find-html-for-class is just a helper function that fills in the arguments for find-location-for-doc-from-buffer for Java buffers.

(defun create-file-list (directory buffer) "Creates the list of files in a directory" (save-window-excursion (let ((default-directory directory)) (shell-command "find . " buffer) (switch-to-buffer buffer) (flush-lines "\.svn") (flush-lines "class-use")))) (defun find-location-for-doc-from-buffer (arg buffer-name buffer-creation-fn begin) "Finds the file for a given documentation name in the buffer
that may be created with buffer-creation" (save-excursion (save-window-excursion (let ((doc-buffer (or (get-buffer buffer-name) (funcall buffer-creation-fn)))) (switch-to-buffer doc-buffer) (goto-char (point-min)) (while (not (line-matches (concat "/" arg "\.html"))) (search-forward arg)) (concat begin (buffer-substring (1+ (line-beginning-position)) (line-end-position)))))))

These next functions are used to look up documentation. my-java-describe-class will open up documentation for the input class file, whereas java-describe-variable will take a variable name and look backwards to it’s declaration and find documentation for that class. c-search-docs does something similar; it will prompt you for a keyword and see if anything in my c++ documentation matches it.

(defun-my java-describe-class "Open Javadoc for Class" "Loads javadoc for specified class in your browser." (interactive "MClass Name: ") (browse-url (java-find-html-for-class arg))) (defun-my java-describe-variable "Open Javadoc for Variable" "Opens the javadoc for the variable at point, if possible." (interactive) (save-excursion (re-search-backward (concat "[ \t\n]" "[A-Za-z]+" "<[][A-Za-z0-9<>]*>" "[ \t\n]" arg)) (forward-char) (java-describe-class (current-word)))) (defun-my c-search-docs "Documentation For" "Searches C++ Documentation for the requested term" (browse-url (find-location-for-doc-from-buffer arg "*C Documentation*" (lambda () (create-file-list "~/.emacs.d/documentation/c++/" "*C Documentation*")) "~/.emacs.d/documentation/c++/")))

Another task that I frequently have to do is fix imports in Java classes. Doing this manually is a huge pain, so I wrote a few functions to help. my-java-import-class will prompt for a class, look up it’s full package name, and add the import to the top of your file. Java-get-undefined-classes will run compile-command and parse the output to add all unimported classes. This needs java-undefined-symbol-regexp to be defined correctly, as well as compile-command to be set to something like ‘javac filename’.

(defun-my java-import-class "Import Class" "Adds an import statement for the class at point." (save-excursion (let ((my-retn-value nil)) (let ((my-string (java-find-html-for-class arg))) (find-file my-string) (end-of-buffer) (re-search-backward "\"\\([A-Za-z0-9]+\\.\\)+[A-Za-z0-9]+ [ci][ln][at][se][sr]" ) (let ((start (point))) (re-search-forward " " ) (setq my-retn-value (substring (buffer-string) start (- (point) 2))))) (kill-buffer (current-buffer)) (beginning-of-buffer) (re-search-forward "import " (point-max) t) (beginning-of-line) (when (looking-at "import") (end-of-line) (newline)) (insert "import " my-retn-value ";\n" )))) (defvar java-undefined-symbol-regexp "symbol : class \\([A-Za-z0-9]*\\)")
(defun java-get-undefined-class-names () (interactive) (save-window-excursion (remove-if #'not (remove-duplicates (mapcar (lambda (x) (if (string-match java-undefined-symbol-regexp x) (match-string 1 x))) (split-string (shell-command-to-string compile-command) "\n *^\n")) :test #'string-equal)))) (defun java-import-undefined-classes () (interactive) (save-window-excursion (mapc #'java-import-class (java-get-undefined-class-names))))

Alex Bennée: Avoiding RSI one keypress at a time

Alex — Wed, 04 Nov 2009 03:04:38 -0800

Much as I love emacs some of the key combinations it expects of you to a) remember b) use are don’t help when your trying to ameliorate the effects of RSI. As I spend an lot of time in compilation mode the following quick win helps:

(define-key compilation-mode-map (kbd "n") 'compilation-next-error)
(define-key compilation-mode-map (kbd "p") 'compilation-previous-error)

If you try executing (self-insert-command) the buffer would complain it’s read-only anyway. I took this single keypress approach to the next level when I was doing a lot of patch merging and tweaked diff-mode to be a little less multi-finger.

Back from vacation

Sun, 27 Sep 2009 22:28:02 -0700

I just got back from a 2-week vacation, so I've been busy catching up on things.

Xcode Requirement for iPhone OS 3.0.1

Fri, 31 Jul 2009 14:20:15 -0700

I've just updated my iPhone OS to 3.0.1 (it's a security upgrade to fix the iPhone SMS vulnerability that was recently announced) and found that I could no longer build/install development apps on my iPhone. A bit of searching uncovered this Apple advisory:
http://adcdownload.apple.com/iphone/iphone_sdk_3.0__final/iphone_os_3.0.1_advisory.pdf

So, if you plan to install iPhone OS 3.0.1 on an iPhone that you use for iPhone app development, you'll need to follow the instructions in the advisory. Note that the terminal command that is described in the advisory is one long line, not 2 lines.

HebrewBible App Store Rankings

Tue, 28 Jul 2009 12:39:03 -0700

I use a variety of different tools to measure how my HebrewBible iPhone App is doing. With the latest 2.0 version of the app, it has now moved into the top 100 of the "Reference" app category in both the USA and France (the top markets for the app). The following MajicRank graph illustrates the current rankings:

Woohoo!

Update-2009-07-28: After 1 week, the app's performance in both the USA and France has improved even more and I've broken into the top 100 for another "Big Eight" country - Canada!:

Woohoo, Woohoo!

New Version of Hebrew Bible iPhone App Available

Sat, 18 Jul 2009 18:47:07 -0700

Today, Apple approved Version 2.0 of my Hebrew Bible iPhone application. In addition to providing 8 different versions of the Hebrew Bible (which can either be read online with no content stored on the iPhone or cached locally for off-network reading) and Hebrew word lookup/lexicon, this version includes links to over 200 Wikipedia articles and provides Google Maps of over 1,000 locations.

Apple made me change the rating of the app to "17+" to indicate that there are "Frequent/Intense" types of "Mature/Suggestive Themes". Their reasoning for this was:

"Our review indicates that the application content is not consistent with the current rating. HebrewBible allows unfiltered access to wikipedia.org, where content with mature or suggestive themes can be accessed."

Come on guys, you've got to be kidding - give me a break! Hmm, a rating of "17+" seems to boost sales of certain movies - maybe I can expect a sudden spike in downloads now! ;-)

Good thing I got my iPhone 3GS last week

Mon, 29 Jun 2009 16:00:25 -0700

It looks like Rogers is all sold out of all models of the iPhone 3GS throughout Canada. Good thing I got mine last week - I've been really pleased by how much better the user experience is with the 3GS over my old iPhone 3G. The features I find most compelling are:

Speed: The increase in processor speed (600MHz CPU as opposed to the 412MHz iPhone 3G) and RAM (256 MB vs 128MB on the iPhone 3G) make a HUGE difference.
Oleophobic Screen: The yucky fingerprint-covered screen of the iPhone 3G is now a thing of the past. The new iPhone's screen coating is the secret and it makes a big difference in the visual appeal of the display.
Video and Camera: The video camera was a nice additon; however, the camera has improved significantly as well. The camera is now 3 megapixels (vs 2 megapixels in the 3G) and both video and camera have geotagging, autofocus, and tap-to-focus functionality. I rarely used the camera on the 3G because of the poor picture quality but I can see myself using the 3GS camera a lot.
Magnetometer: This is most visible in the new Compass app that comes with the 3GS. However, the more useful application of the direction-awareness provided by the magnetometer is in the Maps app. If you press the "location" icon in the bottom-left of the Maps app, it displays your current location (as it did with the iPhone 3G). However, if you press it a second time, the map reorients itself to the direction you're facing and the blue location dot displays a "flashlight" to illustrate the direction you're heading. Very useful when you're in an unfamiliar neighborhood!

New iPhone + New Apple Developer Provisioning Profile

Sun, 21 Jun 2009 21:54:27 -0700

Yesterday, I took some time off from the nice weather and gardening tasks to go out and get a new iPhone 3GS. First off, let me say that the speed difference is really great! I've only played around with a few of the new features (such as the video camera and compass), but the speed increase is a huge plus. Today, I got around to creating a new developer profile for my new iPhone 3GS. My old iPhone was having some problems a few months ago and Apple replaced it, so I remembered the hassles I had then setting up a new provisioning profile for the replacement device (this seems to be a common iPhone developer complaint). I had similar problems this time with the 3GS, so I've decided to note down all the things that I did to get it setup properly. Hopefully, this will help someone else (or, at least, it might remind me of the necessary steps the next time I have to do it!). Not all these steps may be necessary (and it's probably a lot easier if you're just adding a device to an existing profile rather than replacing a provisioning profile); however, I wanted to make sure I knew all the steps this time and it's what worked for me!

When you upgrade to a new device, you need to do the following:

In Apple's iPhone Developer Program Program Portal:

Create a new provisioning profile for the device
Set up a new device profile
Download the new provisioning profile to the Desktop

On the Mac:

Delete the old provisioning profile from ~/Library/MobileDevice/Provisioning Profiles/
Drag the new provisioning profile from the Desktop into ~/Library/MobileDevice/Provisioning Profiles/

In the Organizer window in Xcode:

Click on the device (in the left-hand pane)
Drag the new provisioning profile into the "Provisioning" box
Click on the "Provisioning Profiles" (in the left-hand pane)
Double-click on the new profile name (in the top pane) - it should appear in the "Included Devices" bottom pane with an "Installed" status
Close the Organizer window
Press the Xcode/Empty Caches... menu option
Close Xcode

On the Mac and iPhone:

Power off both the Mac and the iPhone
Power on both the Mac and the iPhone
Attach the iPhone to the Mac

For each project you previously built with the old provisioning profile:

Right-click on the [projectname].xcodeproj file and select "Show Package Contents"
Make a backup copy of the project.pbxproj file
Use a text editor (I used Emacs) to edit the project.pbxproj file:
1. Delete any lines that have the key CODE_SIGN_IDENTITY
2. Delete any lines that have the key PROVISIONING_PROFILE
3. Delete any lines that have the key "CODE_SIGN_IDENTITY[sdk=iphoneos*]"
4. Delete any lines that have the key "PROVISIONING_PROFILE[sdk=iphoneos*]"

In Xcode, for each project you previously built with the old provisioning profile:

Open the project
Press the Build/Clean All Targets menu option
Press the Project/Edit Project Settings menu option
On the Build tab under "All Configurations", chage the "Code Signing Identity" value to iPhone Developer
On the Xcode dropdown menu, select to build on the Device
Press the Build/Build and Run menu option (the application should build and load on the device)

That's what worked for me. If anyone has an easier/alternative approach, let me know and I'll publish relevant comments in updates to this post.

Twin Cities Lisp Group Inaugural Meeting

Fri, 08 May 2009 11:53:01 -0700

Two years ago, Robert Goldman gave a presentation to our Vancouver lispvan group. I'm pleased to note that he's decided to start up his own Lisp User Group in Minneapolis. Here are the details of their upcoming inaugural meeting:

TWIN CITIES LISP GROUP INAUGURAL MEETING 9 June 2009, 6:00 PM - 8:00 PM CDT Common Roots Cafe Corner of 26th and Lyndale, Minneapolis, MN http://preview.tinyurl.com/tclisp-crc Temporary web page: http://rpgoldman.real-time.com/tc-lisp.html
There's a lisp renaissance out there, spurred by the availability of several high-quality open source Common Lisp implementations (SBCL, CMUCL, Clozure CL, and CLISP, at least). After attending the 2009 International Lisp Conference in Cambridge ( http://www.international-lisp-conference.org/2009/index) I was fired up to see if we could share in some of this renaissance here in the Twin Cities.

I was also inspired by visiting with the Vancouver Lisp group, Lispvan. They meet roughly monthly at some location offering coffee, beer, and wine, where they have a talk and some mingling.

In the interests of setting up something like this here, I've (well, Josh Hamell did all the real work) set up a mailman mailing list, tc-lispers ( http://sift.info/mailman/listinfo/tc-lispers_sift.info), and I've reserved the meeting room at Common Roots Cafe ( http://www.commonrootscafe.com/ourcommunity.htm), which offers the aforementioned coffee, beer, and wine (and tasty food --- vegetarian is available and there seem to be at least some vegan options).

At the first meeting John Maraist will talk about the NST unit testing framework for Common Lisp, which he has been developing at SIFT (abstract follows). Then we can plan a next meeting, chat, mingle, and swap stories. Share and learn the latest Lisp applications, techniques, packages, and implementations. Show off your lambda tattoos, evil hacks, and wigflip graphics.

Please come if you can and, whether you can or not, please sign up for the tc-lispers mailing list.

Best,

Robert Goldman
NST: A Unit Test Framework for Common Lisp
In this talk we will introduce the Lisp unit test framework NST. SIFT developed NST for internal use on a number of ongoing Lisp projects, and we believe that the system is now mature enough to release more broadly. In this talk we will review the notion of a unit test, and introduce the use of NST in Lisp project development. NST's implementation makes interesting use of a number of Lisp features --- macro expansion, compile-time execution, metaobject protocols --- and we will discuss its current implementation as well as the lessons we learned along the way.

If you will be in the Twin Cities area on June 9, be sure to attend this new user group meeting. Best wishes to Robert and all the other Twin Cities lispers! By the way, if anyone else is inspired to start a new Lisp User Group, you might want to read my "How to start a Lisp User Group" blog post. I've tried to summarize my experiences in that post.

Lisp/Scheme/Emacs Pipe

The Zipper [PDF]

Zach Beane: Go try XONG

Lispjobs: Software/firmware developer (Lisp), San Diego, CA

Zach Beane: LISP is difficult to learn

François-René Rideau: Software Irresponsibility

Vladimir Sedach: November Montreal Scheme/Lisp User Group presentation

Patrick Stein: Playing with Sheeple-based GUI atop CL-OpenGL

Christophe Rhodes: 19 Nov 2009

Vladimir Sedach: Templates and code generation

Christophe Rhodes: 18 Nov 2009

Andreas Fuchs: Hunchentoot gets a debugging-acceptor

Grant Rettke: Carl Sagan’s Apple Pie

Grant Rettke: Conway’s Law

Programming Praxis: Master Mind, Part 2

Ben Simon: NuGram - an architecture worth pondering

Dominique Boucher: 1-session-per-process - some further comments on the NuGram architecture

Grant Rettke: The Grass is Always Greener

Programming Praxis: Master Mind, Part 1

Llewellyn Pritchard: Experimental monolitic IronScheme executable

Yinso Chen: BZLIB/SESSION.plt - a Session Store via BZLIB/DBI.plt

Yinso Chen: Web Sessions vs. Continuations

Programming Praxis: Two Linear Sorts

Dominique Boucher: The architecture of NuGram Hosted Server

Programming Praxis: Merge Sort

Yinso Chen: Building a Web Session Store (1)

Grant Rettke: Where to Find Jobs in Statistics

Yoni Rabkin Katzenell: CamelHumpSupportForEmacsMovement

Emacs-fu: showing pop-ups

showing pop-ups from org-mode appointments

showing pop-ups for new mail

Got Emacs?: Churn baby, Churn

Random Sample: Posting org buffers with weblogger.el to your blog

Chris Ball: Btrfs snapshots proposal

Random Sample: Posting Drupal blog entries from within Emacs

Matt Harrison: New insight on 100% coverage

Bryan Murdock: Initial Thoughts on Google Go

Yoni Rabkin Katzenell: There is Very Little Good on the Web; Keep it Alive

sachachua: Automating tedious wiki editing tasks with Emacs and w3m

Alex Bennée: Thinking about email

Joseph Miklojcik: Lexical Analysis

Joseph Miklojcik: jJava: a possible Phosphoros competitor?

Yann Hodique: https traffic logger

Yann Hodique: s5 presentation from org-mode

Thomas Kappler: Announcing md-readme

Justin Heyes-Jones: Transparent emacs on windows

Phil Hagelberg: in which things are mapped, but also reduced

Emacs-fu: copying lines without selecting them

Flickr tag 'emacs': Tidy up clojure code in TextMate

Flickr tag 'emacs': Tidy up clojure code in TextMate

Bryan Murdock: I Got a Mac

Got Emacs?: Another Day, Another Language: Go: From Google

Phil Jackson: A Magit clone

Matt Keller: Switching to Atom from RSS

Thomas Kappler: Now focus!

@emacs: emacs: Happy 3 years of Planet Emacsen! http://planet.emacsen.org/

Edward O'Connor:

Nathaniel Flath: Java and C++ Utilities

Alex Bennée: Avoiding RSI one keypress at a time

Back from vacation

Xcode Requirement for iPhone OS 3.0.1

HebrewBible App Store Rankings

New Version of Hebrew Bible iPhone App Available

Good thing I got my iPhone 3GS last week

New iPhone + New Apple Developer Provisioning Profile

Twin Cities Lisp Group Inaugural Meeting

showing pop-ups from `org-mode` appointments