Accidentally Discovering a Password Generator in Perl 6

Would you like a random number?

say rand;
# 0.468924762180761

How about between 0-10?

say 10.rand;
# 9.18202878042811

Let’s cut that down to an integer

say 10.rand.Int;
# 7

Let’s create an array of things from a range

say ["c"..."m"];
# [c d e f g h i j k l m]

How about we pick one of those

say ["c".."m"].pick;
# f

Suppose we don’t want to fill an array with a big range. Suppose we just want to represent a range, and pick from it without having to allocate that whole array

say pick 5, "0".."z";
# (y e E < m)

Pick, once it’s picked something, will never pick it again. It’s picked, and it’s gone. But if you want to do more like rolling dice and get repeats, use roll instead of pick

say roll 5, "0".."z";
# (F c = o =)

See how the output is in parenthesis? That’s because it’s returning a sequence

say (roll 5, "0".."z").WHAT;
# (Seq)

You can iterate over those if you like, or do whatever. Or you can join them into a single string

say (roll 5, "0".."z").join;
# OMdYO

That’s looking suspiciously familiar. How about we generate a random number of rolls

say (roll 10.rand.Int, "0".."z").join;
# _JJF

Oh dear that’s not very many. Let’s say we have to have at least 12!

say (roll 10.rand.Int + 12, "0".."z").join;
# pCKtVB4j=^ShIZHyI

How about we have 10 of those to choose from?

for 0..9 {
    say (roll 10.rand.Int + 12, "0".."z").join;
}
# tEApXfu<klX<yn;Z6NXD4
# =n?GLn1QG3JBD2r_vz
# X49tYWtSap`H
# gyS[7_5Y5?w^
# j>oD4A4iTixa_87wVq
# ltSn<rDEM7Z^
# eyU_d?49^1Z:7;I=?1l
# nIFo2ZF20>l5j>Ord9
# cfKrtqjt3XG<\1G2o=
# TeJnVgy2qyJRBlB

Or if you wanted to, maybe you’d prefer, up to 10 😉 (and some other clarity)

for ^10 {
    say (roll (10.rand.Int + 12), "0".."z").join;
}

 

A Perl Grammar Tutorial ( Beginner’s Guide )

This tutorial became part of the official Perl 6 documentation as Grammar Tutorial where it became a “living” document which can be edited and expanded upon by others. As such, this document may be out of date, for better or for worse.

Why Grammars?

You will always run into string parsing that will give you a headache. It it said that HTML, for example, cannot be broken down and parsed effectively at all, simply by using regular expressions to sort out the elements. Another example would be defining the order in which words and symbols might make up a language and provide meaning. This is when Perl’s grammar system fits perfectly.

When Would I Use Grammars?

Grammars are great at taking stings, trying to make sense of them, and then saving them into a data structure you can actually work with. If you have strings of some kind that you need to bring order to, or interpret, grammars give you some great tools to make it easier.

Your string might be a whole file that you need to break into sections. Or perhaps line by line. Maybe you have a protocol like SMTP you’re working with, and want a convenient and organized way to define which “commands” need to come after what data from the user to make the protocol work. Maybe you’re wanting to create your own string-based protocol. Maybe you’re designing your own language.

The Broad Concept of Perl Grammars

Regular Expressions (regex) work well to find patterns in strings and manipulate them. However, when you need to find multiple patterns at once, or need to combine patterns, or test for patterns that may surround strings, or other patterns – regular expressions alone are not adequate.

Grammars provide a way to define how you want to examine a string using regular expressions, and you can group these regular expressions together to bring even more meaning.

For example, in the case of HTML, you could define a grammar that would recognize HTML tags, both the opening and closing elements, and the text in between, and act upon these elements as a whole by stuffing them into data structures, such as arrays or hashes, that you can then easily use. In essence, Grammars provide a means to define an entire language or specification that can be used to parse strings of arbitrary sizes and complexity.

Getting More Technically into Grammars

The conceptual overview

Grammars are defined as an object, just like everything else in Perl. Technically, they are normal classes with a little extra magic thrown in, which we’ll get to later — and a few limitations. You name and define a grammar exactly as you would a class, except using the “grammar” keyword instead of “class”.

grammar My::Gram { ..methods 'n stuff... }

Grammars contain elements like methods, called regex, token or rule. These elements are named, just like methods are named. And each one defines a regex, token or rule (which are mostly the same thing (not really) but we’ll get to that later).

Once you have your grammar defined, in your program call it by name and pass in the string you want to parse, and that string will be run through the rules you defined based upon your regex, token and rule “methods”.  When done, you get back a Match object that has been populated with data structured and stored by the names you used to define your methods.

my $matchObject = My::Gram.parse($what-a-big-string-you-have);

Now, you may be wondering, if I have all these regexes defined that just return their results, how does that help with parsing things that may be forwards or backwards in a string, or things that need to be combined from multiple of those regexes… and that’s where grammar actions come into play.

For every “method” you match in your grammar, you get an action you can call to do something funny or clever with that match. You also get an over-arching action you can use to tie them all together and custom build a data structure you might want to return, where all your crazy string parsing makes sense in your nicely ordered and defined data structure. This over-arching method is called TOP by default. We’ll get more into this as well.

The technical overview

Grammars are defined just like a class, but using the grammar keyword in place of class. The “methods” in grammars are called either regex, token or rule. Regex methods are slow but thorough — they will look back in the string and really try. Token methods are much faster and they ignore whitespace. Rule methods are the same as token methods except that they pay attention and consume whitespace in your “regex” definitions.

When a method (regex, token or rule) matches in the grammar, that matched string is put into the Match object that will eventually be returned, and it will be keyed with the same name as the method you chose to name it.

grammar My::Gram {
    token TOP { <thingy> .* }
    token thingy { 'clever_text_keyword' }
}

So in this, if you were to my $match = My::Gram.parse( $string ) — and your string started with the characters ‘clever_text_keyword’, you would get a match object back that contained ‘clever_text_keyword’ keyed by the name of ‘thingy’ in your match object. These can grow much more complex, as your needs require, as you might imagine.

Now, to mention TOP. The TOP method (regex, token or rule) is the overarching regex that must match everything (by default). If the string you pass in to parse doesn’t match the TOP regex, your returned match object will be empty (Any).

As you can see above, in TOP, the “<thingy>” token is mentioned. The <thingy> is defined on the next line, “token thingy…”. That means that ‘clever_text_keyword’ must be the first thing in the string passed in, or the grammar parse will fail, and we’ll get an empty match. This is great for recognizing malformed stuff that someone might give you that should be thrown away.

Learning By Example – a REST Contrivance

Let’s suppose we’d like to parse a URL into the component parts that make up a RESTful request. Let’s decide that we want the URL’s to work like this:

  1. The first part of the URI we’ll call the “subject”, like a part, or a product, or a person.
  2. The second part of the URI we’ll call the “command”, like standard CRUD stuff (create, retrieve, update, or delete).
  3. The third part of the URI will be arbitrary data. Perhaps the specific ID we’ll be working with, or a long list of data separated by “/”‘s.
  4. When we get a URL, we’ll want 1-3 above to be placed into a nice data structure we can use without having to do all sorts of splitting, and that can be easily altered in the future or expanded upon (or extended).

So if we got a URI on the server of “/product/update/7/notify” we would want our grammar to give us a nice $match object that has a “subject” of “product”, a “command” of “update” and “data” of “7/notify” (for now).

The first thing we do is define the grammar class. We’re going to need to define our subject, command and data as well. I think we’ll use token for them, since we don’t care about whitespace in the regex.

grammar REST {
    token subject { \w+ }
    token command { \w+ }
    token data    { .* }
}

So far this REST grammar says we want a subject that will be just word characters, a command that will be just word characters, and data that will be everything else left in the string (URI in this case).

But in our big string we get, we don’t know what order these regex matches will come in. We need to be able to place these matching tokens in the larger context of our URI we’ll be passing in as that string. That’s what the TOP method is for. So we add it, and place our tokens by name within it, along with however else our valid string should look, coming in.

grammar REST {
    token TOP     { '/' <subject> '/' <command> '/' <data> }
    token subject { \w+ }
    token command { \w+ }
    token data    { .* }
}

You could actually use this to extract your data from the URI for basic CRUD that has all 3 parameters included:

my $match = REST.parse('/product/update/7/notify');
say $match;

「/product/update/7/notify」
 subject => 「product」
 command => 「update」
 data => 「7/notify」

Of course, the data can be accessed directly by using $match<subject> or $match<command> or $match<data> to return the values parsed. They each contain match objects you can work further with, or coerce into a string ( $match<command>.Str )

Adding some flexibility

The REST grammar so far will handle retrieves, deletes and updates ok. However, a create command doesn’t have the third part (the data portion). This means our grammar will fail to match if we try to parse an update URL, and everyone will scream. To avoid this, we need to make that last data position match optional, along with the ‘/’ preceding it. This is easily accomplished by adding two question marks to the TOP token, to indicate their optional nature, just like a normal regex. So now we have:

grammar REST {
    token TOP     { '/' <subject> '/' <command> '/'? <data>? }
    token subject { \w+ }
    token command { \w+ }
    token data    { .* }
}

my $m = REST.parse('/product/create');
say $m<subject>, $m<command>;

# 「product」「create」

Let’s imagine, for the sake of demonstration, that we might want to allow these same URI’s to be entered in by a user from the terminal. In that case, they might put spaces between the ‘/’s, since users are prone to break things. If we wanted to accommodate this possibility, we could replace the ‘/’s in TOP with another token that allowed for spaces on either side of it.

grammar REST {
    token TOP     { <slash><subject><slash><command><slash>?<data>? }
    token subject { \w+ }
    token command { \w+ }
    token data    { .* }

    token slash   { \s* '/' \s* }
}

my $m = REST.parse('/ product / update /7 /notify');
say $m;

###
「/ product / update /7 /notify」
 slash => 「/ 」
 subject => 「product」
 slash => 「 / 」
 command => 「update」
 slash => 「 /」
 data => 「7 /notify」

We’re getting some extra junk in our match object now, with those slashes, but there are some very nice ways to make a tidy return value that we’ll get to.

Adding some constraints

We want our RESTful grammar to allow for CRUD operations only. Anything else we want to fail to parse. That means our “command” above should have one of four values: create, retrieve, update or delete.

There are several ways to accomplish this. For example, you could change the command method:

token command { \w+ }
    
  ..becomes..

token command { 'create'|'retrieve'|'update'|'delete' }

This results in any URI coming in getting checked; where the second string between ‘/’s must be one of those values, or the parsing of the URI fails. Exactly what we’d like.

There is another way, though, that can give you far greater flexibility to do interesting or unholy things with your regexes, and provide some better readability when options grow large. These are proto-regexes.

To utilize these multimethods (here called protoregexes) to constrain our command to the same values we had above, we’ll replace “token command” with the following:

proto token command {*}
token command:sym<create>   { <sym> }
token command:sym<retrieve> { <sym> }
token command:sym<update>   { <sym> }
token command:sym<delete>   { <sym> }

Using protoregexes like this gives us some greater flexibility. For example, instead of returning <sym>, which is the string that was matched, we could instead enter our own string, or do other funny stuff. We could do the same with the “token subject” method, and limit it also to only parsing correctly on valid subjects (like ‘part’ or ‘people’, etc.).

Putting our RESTful grammar together so far

This is what we’ve come up for processing our RESTful URIs so far:

grammar REST
{
    token TOP { <slash><subject><slash><command><slash>?<data>? }
    
    proto token command {*}
    token command:sym<create>   { <sym> }
    token command:sym<retrieve> { <sym> }
    token command:sym<update>   { <sym> }
    token command:sym<delete>   { <sym> }
    
    token subject { \w+ }
    token data    { .* }
    token slash   { \s* '/' \s* }
}

Let’s look at various URI’s and how they behave being passed through our grammar.

my @uris = ['/product/update/7/notify',
            '/product/create',
            '/item/delete/4'];

for @uris -> $uri {
    my $m = REST.parse($uri);
    say "Sub: $m<subject> Cmd: $m<command> Dat: $m<data>";
}

###
Sub: product Cmd: update Dat: 7/notify
Sub: product Cmd: create Dat: 
Sub: item Cmd: delete Dat: 4

So with just this part of a grammar, we’re getting almost everything we need. Our URI’s get efficiently parsed and we’re given a nice little data structure for the variables we need to work with.

But look at that first line returned — the data token is returning the entire end of the URI as just one string. We need to be able to work with that 7 there. And that 4! Well, the 4 is easy… But the 7 had the extra /notify on the end, to signal the system to notify someone that a product was updated (perhaps).

So let’s make sure we can do stuff with our regex tokens that were matched, such as that data token that returned a “7/notify”. And to do so, we’ll take advantage of another characteristic of these Grammar classes — a thing called actions.

Grammar Actions

We’re going to diverge from our example for a moment to talk about Perl’s grammar actions. We’re going to do this because, in many ways, grammar actions are separate from the grammars you define. They are a completely different class that you create, and use from your grammars to do stuff with the matches you find in your grammars.

You can think of grammar actions as a kind of plug-in expansion module for grammars. A lot of the time you’ll be happy using grammars just on their own. But when you need to process some of those strings further, you can plug in the expansion module.

You do this when you first create your grammar. In addition to passing in the actual string you want to parse, you can pass in a named parameter called “actions” which should contain an instance of your actions class. From our example above, if our actions class were called REST-actions we would parse our grammar string like this

my $matchObj = REST.parse($uri, actions => REST-actions.new);

   ...or if you prefer...

my $matchObj = REST.parse($uri, :actions(REST-actions.new))

Grammar actions get pulled into your grammar in this way. And once there, your grammar can utilize the actions you define. And those defined actions are defined as normal methods in your action class.

The only weird bit is that if you name your action methods with the same name as your grammar methods (tokens, regexes, rules), then when your grammar methods match, your action method with the same name will get called automatically for you, and it will be passed the match object from that specific grammar token that matched.

In other words, when you’ve attached an action class, name the methods in that class with the same names you used in your grammar class, if you want actions to be called automatically when grammar regexes match.

Matching actions will get passed the Match object from the grammar token as its argument, and this can be represented by the $/ variable. This means that all your action methods should take $/ as the argument, then you get to work with that grammar match object in your action. So much for words, let’s return the example scenario to try and find more solidified clarity.

Grammars by Example with Actions

Here we are back to our grammar, as we left it.

grammar REST
{
    token TOP { <slash><subject><slash><command><slash>?<data>? }
    
    proto token command {*}
    token command:sym<create>   { <sym> }
    token command:sym<retrieve> { <sym> }
    token command:sym<update>   { <sym> }
    token command:sym<delete>   { <sym> }
    
    token subject { \w+ }
    token data    { .* }
    token slash   { \s* '/' \s* }
}

Now we want to make sure that our data token gets parsed a little bit further, so that we can separate out an ID if we get it, from the rest of the long URL that might follow, such as “7/notify” in our example.

To accomplish this we’ll create an action class, and in it, create a method with the same name as the named token, rule or regex we want to process. In this case, our token is named “data”.

class REST-actions
{
    method data($/) { $/.split('/') }
}

Now when we pass the URL string through our grammar, the “data” token match will be passed to the action class (REST-actions) to the method “data”, and we’ll split that URI string by its ‘/’ character. That way, the first element of the returned list will be our ID number (7 in the case of “7/notify”).

But not really.

Keeping grammars with actions tidy with “make” and “made”

If our grammar calls our action above on data, the data method will be called, but nothing will show up in the big TOP grammar match result returned to our program. In order to make our action results show up, we need to call “make” on that result, and that result can be many things, including strings, array or hash structures.

You can imagine that the “make” we put on our action results, places that result in a special, contained area in our whole grammar. Everything that we “make” for data structures, can be accessed later by “made”.

So instead of our REST-actions class above, we should write

class REST-actions
{
    method data($/) { make $/.split('/') }
}

When we add “make” to our match split (which returns a list), our action will return a nice data structure to our grammar that will be stored separately from the “data” token of our original grammar. This way, we can still work with both if we need to.

Now if we want to access just our id of 7 from that long URL, we can access the first element of the list returned from the “data” action we “made” with “make”:

my $uri = '/product/update/7/notify';

my $match = REST.parse($uri, actions => REST-actions.new);

say $match<data>.made[0];  # 7
say $match<command>.Str;   # update

Here, we call “made” on our data, because we want the result of our action that we “made” (with “make”), to get our split array. That’s lovely! But, wouldn’t it be lovelier if we could “make” a friendlier data structure that contained all of the stuff we want, rather than having to coerce types and remember arrays?

Well, just like TOP in our grammar that over-arches and matches the entire string, our actions have a TOP method as well. We can “make” all of our individual match components, like “data” or “subject” or “command” and then we can place them in our perfect fantasy data structure which we will “make” in TOP. When we return our final match object from our grammar processing, we can then access this perfectly-imagined data structure that we’ve cobbled together, to make everyone’s life happy and easier.

To do this, all we have to do is add the method TOP to our action class, and in that method, “make” whatever data structure we like from our component pieces. So, our action class now might become

class REST-actions
{
    method TOP ($/) {
        make { subject => $<subject>.Str,
               command => $<command><sym>.Str,
               data    => $<data>.made }
    }

    method data($/) { make $/.split('/') }
}

Here in our TOP method, our “subject” remains the same as the subject we matched in our grammar. Also, our “command” returns the valid <sym> that was matched (create, update, retrieve, or delete). Each we coerce into .Str as well, since we don’t need the full match object.

But what we want to be certain to do, is to use the “made” method on our $<data> object, since we want to access that split one that we “made” with “make” in our action, rather than the proper $<data> object.

After we “make” something in the TOP method of a grammar action, we can then access all of our custom-made stuff by calling the “made” method on our grammar result object. From our earlier example, it now becomes

my $uri = '/product/update/7/notify';

my $match = REST.parse($uri, actions => REST-actions.new);

my $rest = $match.made;
say $rest<data>[0];   # 7
say $rest<command>;   # update
say $rest<subject>;   # product

Of course you could also shorten it if you knew you wouldn’t need the full grammar return match object, and only wanted your custom-made data from your action’s TOP.

my $uri = '/product/update/7/notify';

my $rest = REST.parse($uri, actions => REST-actions.new).made;

say $rest<data>[0];   # 7
say $rest<command>;   # update
say $rest<subject>;   # product

Oh, did we forget to get rid of that ugly array element number? Hmm. Let’s make a new thing in our grammar’s custom return in TOP… how about we call it “subject-id” and have it be set to element 0 of <data>.

class REST-actions
{
    method TOP ($/) {
        make { subject    => $<subject>.Str,
               command    => $<command><sym>.Str,
               data       => $<data>.made,
               subject-id => $<data>.made[0] }
    }

    method data($/) { make $/.split('/') }
}

Now we can do this instead

my $uri = '/product/update/7/notify';

my $rest = REST.parse($uri, actions => REST-actions.new).made;

say $rest<command>;    # update
say $rest<subject>;    # product
say $rest<subject-id>; # 7

And this is the grammar and grammar actions that got us there, to recap:

grammar REST
{
    token TOP { <slash><subject><slash><command><slash>?<data>? }
    
    proto token command {*}
    token command:sym<create>   { <sym> }
    token command:sym<retrieve> { <sym> }
    token command:sym<update>   { <sym> }
    token command:sym<delete>   { <sym> }
    
    token subject { \w+ }
    token data    { .* }
    token slash   { \s* '/' \s* }
}
class REST-actions
{
    method TOP ($/) {
        make { subject    => $<subject>.Str,
               command    => $<command><sym>.Str,
               data       => $<data>.made,
               subject-id => $<data>.made[0] }
    }

    method data($/) { make $/.split('/') }
}

Hopefully this has helped introduce you to the concept of grammars in Perl and how grammars and grammar action classes can tie together. For further information, insights and oddities, check out the more advanced Perl grammar guide.

Perl 6 Variable Types and Modules

I’m still learning a great deal about Perl and its typing. One of the things you hear a lot is that, in Perl 6 everything is an object. I’m finding this to be true, and it has some oftentimes surprising results.

For example, I’ve recently been playing with the DBIish module which affords database connectivity. We all know that working with date and time types with database columns can be a real pain, getting the formatting of the strings to insert correctly, and interpreting those when selecting out.

Perl has a module called DateTime built in. You can say it’s a module. You can say it’s a data type. You can say it’s an object. PostgreSQL has a timestamp with timezone data type (amongst others).

In my data model class, I defined an object attribute as a DateTime data type, just to play around and see what would happen.

class Shoe {
    has Pkey     $.id;
    has Str      $.description is rw;
    has DateTime $.created is rw;

    method retrieve(Pkey :$id = $.id) {
        ...
        $.created = %row<created_timestamp>;
        say "Created in the year {$.created.year}";
        say "warranty expires {$.created.later(:2years)}";
        ...
   }

You don’t need to worry about creating a DateTime object, other than specifying that your class attribute will be a DateTime type. All the conversions to string, whether TO the database or FROM the database, get handled seamlessly.

Honestly, I didn’t expect this to work — but I’m very happy to never have to worry about date and time formats again, when dealing with database column types! 🙂

Perl 6 Lies – What Might Bite the Newly Arrived

Don’t misunderstand; Perl 6 is wonderful and I am having the best time playing with it – getting used to it beauty and strangeness. This article is to document some unexpected gotchas that got me along the way.

Optional signature parameters are sometimes not optional

One of the nice things you can do with Perl is create subroutines and class methods with parameter signatures that can take care of a lot of tedious coding for you. For example, suppose you want to create a method that changes a shoe in your database table. Your method signature might look like this:

class Shoe {

method update(Int :$id!      where { $id > 0 },
              Int :$color-id where { $color-id > 0 },
              Str :$name! ) {

... method code here ...
}

These are named parameters here, where you could call $shoe.update( id => 23, color-id => 8, name => ‘Hush Puppy’);

Parameters with a “!” are required. Named parameters lacking a “!” are optional, or you could put a “?” there. The “where” clause makes sure that whatever is entered will be constrained by the where clause.

Now, we have color-id being optional. However, if you leave it out, your program will bomb out with the following worthless error message:

Invocant requires an instance of type Int, but a type object was passed. Did you forget a .new?

This message goes away if you include the color-id parameter, however. So, this optional parameter actually is not optional.

But if you ask the Perl guys, they’ll say it is optional. It’s just that when using a “where” clause, a bit more is going on. If we didn’t have that where clause on color-id, it wouldn’t matter.

The issue is, Perl wants to use these where clauses in the signature to make sure that parameters are always constrained and typed correctly — and this includes parameters other than the one you’re checking. That is, your color-id parameter can also check stuff with your id or name parameters as well on invocation. So they have to bind early on, if we want to have that extra bit of wonderful functionality and consistency.

Normally you won’t have to worry about this. The only time it matters is when you’re using an optional parameter that includes a “where” clause in the signature. When you do that, the fix is easy, and makes good sense. In the case above, the color-id line would be changed to the following:

Int :$color-id where { not defined $color-id or $color-id > 0 }

The trick is just remembering: when using an optional parameter with a where clause, make sure the where clause will evaluate to true for the constraint — because that constraint will be checked even when the parameter is marked optional. And the documentation has been updated to make this somewhat more clear.

Using submethod BUILD and using attributes with class inheritance

This is very much like the same problem just described, when optional parameters need to be checked for existence when using the “where” clause in the signature.

Whenever you use a submethod BUILD in a class, you have full control over attributes that are assigned to that class. This also means that you have the responsibility of assigning them.

A great Perl shortcut is to have attributes automatically assigned by specifying them in the signature like this:

submethod BUILD(PosInt :$!id, :$!name, :$!address) {}

This lets attributes be set upon the .new invocation, just as normally happens, which is very handy. However, if you do something like use a custom type, like PosInt above, which expects integers greater than 0, you will get an error about trying to set $!id, even when you didn’t set it in .new(). It will complain about not meeting its type constraints, even though you didn’t set it to anything.

So, just like signatures using where clauses, if your variable is going to be undefined, you need to handle that in the body of the BUILD function and not in the signature — otherwise the auto-whatever assignment system will set the value to Any — which is not a PosInt. 😉 So don’t let that bit do the nice auto-assign stuff.

Once can happen twice, or thrice or more!

Perl has long been known for its powerful and creative control structures. One of my favorite new ones in Perl6 is the “once” control structure.

What’s neat is that it takes care of something we all often have to deal with manually. When you’re doing something in a loop, sometimes you’ll want to do things only once inside it, and after that once, though the loop continues, you won’t do that one-time thing again.

until $database.failure {
    once { say "Starting...";
           $log.writesys: "Starting..."; }

    ...all my normal stuff...
}

This will loop and do your bidding until $database.failure returns True (or whatever). However, the code in the “once” block will only be executed one time, no matter how many times the loop happens.

This very, very handy. But Perl 6 is also very multi-threaddy. So you cannot count on “once” to do something just once if you’re using multiple threads for your task. The variable state stuff just isn’t there to cope with such things.

The documentation for once is also updated with this information now, and yet here is another warning. 🙂

Parsing Command Line Arguments in Perl 6

Like many people, I do a lot of automation using Perl. Years ago I began making utilities more generalized than specific, so I could re-use them in slightly different situations.

This means using a lot of command line arguments passed to Perl so that it knows what to do. Always in Perl 5 I used Getopt::Long to parse command line parameters – a truly great module that was always there. Not so with Perl 6 – some very good command line argument parsing is built in.

The MAIN Subroutine

The key is discovering that Perl 6 supports the use of a subroutine named “MAIN” which is called whenever you run a Perl 6 program, much like a C program. The command line arguments can be captured and processed in the definition of the MAIN subroutine.

For example, say you were writing a utility to crop an image. You want to be able to call it like

cropimage --top=10 --bottom=15 --left=50 --right=30 picture.png

In Perl 6, this is all you would need to do:

#!/usr/bin/env perl6

sub MAIN ( $filename,
           :$left, :$right,
           :$top,  :$bottom ) {

    say "$left $right $top $bottom $filename";
}

Variables get automatically defined in the MAIN scope, so you don’t have to use “my”. And if you put a colon “:” before the variable name, Perl will treat that as a named parameter. Notice how $filename doesn’t have a “:”? So we don’t have to put a named parameter in the command line.  But the other variables have “:”‘s before their names, so that means they have to be a named parameter.

Worth noting as well that Perl seems to like having the non-named parameters come before the named ones in the sub MAIN definition. In other words, positional ones come first, then do named ones.

Optional and Required Parameters

If you want to make a parameter required, shout it and put an “!” after it’s name. If you want it to be optional, put a “?” after it:

sub MAIN ( :$left!, :$right?,  ....

would mean that the named parameter –left MUST be supplied or Perl will generate an error for the user, while the –right parameter is optional.

Parameter Default Values

Now, with our example, we may not need all of the parameters every time, so they should be optional. But we will also want to have default values of 0, since when the parameters are not specified, we’ll want to crop 0 pixels off in that direction.

So lets make all of our parameters optional and with a default value of 0. Of course, parameters with default values are always optional. It’s just a matter of setting an equality in the MAIN definition:

#!/usr/bin/env perl6

sub MAIN ( $filename,
           :$left = 0, :$right  = 0,
           :$top  = 0, :$bottom = 0 ) {

    say "$left $right $top $bottom $filename";
}

With this, we’ll get actual 0’s coming out in our “say” 😉 even if we pass in no parameters but $filename.

Constraining Parameters into Sanity

You know there will always be some smart aleck or some universal happenstance that will throw wrenches at you. It’s the curse of being our own form of order within other’s entropy. Take for example the guy who wants to crop 10.88242 pixels in on the left. Yeah, that guy.

This is where Perl’s variable type system can help. There are LOTS of types available. We’ll keep this simple. In our case, we don’t want to slice up our pretty monitor’s pixels into .88242 segments. We want whole pixels or nothing! An integer, in other words.

To constrain parameters to a type, just place the type name before the variable/parameter definition. You don’t have to, of course, but if you choose to, it can help sometimes.

#!/usr/bin/env perl6

sub MAIN ( Str $filename,
           Int :$left = 0, Int :$right  = 0,
           Int :$top  = 0, Int :$bottom = 0 ) {

    say "$left $right $top $bottom $filename";
}

But you know they’ll be right back, breaking it again, because they thought it would be cute to crop a negative amount of pixels from the image. So maybe you’d want to use the UInt type instead (Unsigned Integer) so the value will have to always be positive.

Another useful thing is specifying a type Bool for a parameter variable, which basically makes it into a flag.

Oh, did I mention you can use dashes “-” in variable names in Perl 6? That amazed me, along with the fact you can use full UTF8 charactersets.

So suppose we wanted to be able to specify an optional output filename for our cropped image.  We’d like something like this to work:

cropimage --left=25 --output-file=cropped.png filename.png

All we have to do is call the named parameter variable $output-file and make it optional.

#!/usr/bin/env perl6

sub MAIN ( Str $filename,
           Int :$left = 0, Int :$right  = 0,
           Int :$top  = 0, Int :$bottom = 0,
           Str :$output-file? ) {

    say "$left $right $top $bottom $filename";

    say "Writing " ~ ($output-file || $filename);
}

Bonus Error Messages!

If you tried any of these and got an error, you’ll see how nicely Perl formats these when you’re using the built-in parameter handling. It’s an actual detailed “Usage:” description that is given to the end user, detailing all the parameters and their types. Very Cadillac. 🙂

Installing Perl 6 on Debian or Ubuntu

Keep in mind, this is written on 09-Sep-2016 and stuff can change over time.

Perl 6 is developing rapidly, specifically, the MoarVM and Rakudo which comprise the environment that implements the Perl 6 specifications.

Every Linux distribution comes with Perl. Many have Perl 6 now as well. With Perl 6’s rapid development, these distribution packages of Perl 6 can be outdated.

This is how I installed Perl 6 in Ubuntu and this should also work for Debian. We rely on a system much like Perlbrew, called Rakudobrew to automatically download and compile the necessary stuff for us, and change our environment to be able to find perl6

Git Your Compile Tools

First, you need to be certain you have the tools and libraries necessary to compile stuff on your local machine.

apt-get install build-essential git

This will install the gcc compiler and various libraries along with the git version control system so we can download the latest rakudobrew and keep up to date over time.

Download and Install Rakudobrew

git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew
 echo 'export PATH=~/.rakudobrew/bin:$PATH' >> ~/.bashrc
 source ~/.bashrc

The first line uses git to download the rakudobrew archive into your home directory in .rakudobrew. No biggie. These instructions are taken (mostly) directly from the Rakudo site.

The second line alters your shell environment PATH to include this directory’s “bin” subdirectory. This is wildly dangerous unless you trust absolutely where you are downloading from, which is never a good idea. You can always just compile it yourself if you follow the instructions on the above-linked Rakudo site.

The third line alters your currently-existing PATH in your terminal to add that .rakudobrew/bin directory (you could also close your terminal and open a new one instead).

Have Rakudobrew Compile Rakudo and MoarVM

This bit’s the easy part now, thanks to Rakudobrew.

rakudobrew build moar
rakudobrew build panda

The second line builds and installs Panda, which seems to be a cpanm-like thang for Perl 6 modules. To have a complete Perl 6 experience, you’ll need some of those modules that are commonly bundled up with Perl. You can get them with

panda install Task::Star

And they’re you’ll have Rakudo Star! Like cpan or cpanm, it will take a while to download and compile the modules.

How to Run a Perl 6 Program

Easy enough if you’ve done the bit above to include .rakudobrew/bin in your shell path. You can just type “perl6” and have an interactive interpreter to play around with.

Or if you prefer to edit a file, you can run the file with

perl6 filename

Tough, eh? Or if you’d rather the OS knows to do perl6 for you, give it that magic hashbang at the top of the file like so

#!/usr/bin/env perl6
my $code = "about normal";say "I know nothing " ~ $code;
("I know nothing" ~ $code).say;
say "The new $code";

No need for warnings. No need for use Modern::Perl . No need for strict, Try::Tiny, etc.

There ya go!

Perl 6 First Impressions

Perl Camelia LogoStarting at the beginning in Perl6, I had no clue. I only have a slight one now. But I’ll be documenting some of the things I learn in subsequent posts, in case it can help anyone in the future.

This first post will be about initial impressions, coming from someone who knows a bit about earlier Perls, and has held off actually putting Perl 6 into work.

Background Info

Perl now runs via a VM – right now either the standard JVM or a new one called MoarVM (which people seem to like best and is default, and which I like too because it hasn’t hurt me yet).

Perl itself is a language specification. It relies on the Rakudo compiler to implement that language specification and tie it into MoarVM. This is a great separation of concerns — MoarVM can worry about interacting with the operating system, Rakudo can worry about what it needs to tell MoarVM to do and what the heck a Perl coder meant to happen, while Perl itself remains only in the abstract.

Why Use Perl6 when Other Languages Work Great?

For me it was curiosity. I love coding in Perl, for the most part. It’s the most flexible and powerful language I’ve encountered. I’ve been curious about Perl6 for years, but resisted putting any of it into production until recently. There’s something about it now that feels solid and ready. Here are some of the things that feel compelling to me right now.

Close to the OS

One of the several reasons I reach for Perl before any other language is because it runs close to the OS. Perl grew out of shells and shells grew out of Perl. I was worried I might lose this with Perl6. So far, I am very happily surprised. There is much that makes interacting with the shell/OS even easier – things that don’t require finding the module du jour and reading or refreshing up on it. It just flows in and out of the language.

Bondage or Freedom

It’s your choice! Perl has traditionally been very flexible, and many people who like to be told exactly what to do, or like to tell others exactly what to do — these people become easily exasperated by the freedom of expression Perl offers.

Eventually Perl modules were created that acted like tight girdles and cat-o-nine-tails upon those people, to help satisfy them. Now Perl6 has its own build-in dungeon basement!

You don’t have to use it though. But it’s there if you want it, and it’s seamlessly part of everything.

For example, you can enforce variable types if you want to, and a great many of them too, in bizarrely interesting ways. And you can coerce them around. Or you can leave everything all free-form.

You can use positional parameters when calling methods or subroutines, enforce types there too, or use named parameters, or just sloppily slurp in data. (or neatly slurp 😉 )

And there are many ways to die, even in an orderly way with exceptions. And the default error messages are pretty great, most of the time, with calling stacks included by default, without having to use modules.

And control structures… well, there’s a veritable zoo. Or just your standard, well-known tried and true for those who shrink at variety.

Perl is Smart

Perl has never been limited to being discreetly-sized Lego blocks that kids can easily make a toy house from. It can be used that way, of course. And often it is, by people new to the language.

For me, Perl’s greatest strength is that it has always allowed me to work very intimately with the structure, flow, manipulation and presentation of data, in whatever form, as an actual capability built into the language itself — and to accomplish it in ways that are both optimal and intuitive.

These capabilities have only increased with Perl6 as far as I have seen, though some semantics for getting there have changed. But those changes I believe are justified because I can see how they make sense, to give room to other capabilities that have been added, and to provide some clarifications.

In case that doesn’t make sense, one little example is that string concatenation is no longer done with a dot “.”. In Perl6, variables (objects) can have a dot after them to call a method on that object — that could potentially get confused with a concatenation dot. So they’ve made the concatenation into a tilde “~” instead. It actually looks better I think, but it was a nightmare to figure out when first starting to play. I’ll be documenting some more gotchas as I run across them.

BTW – that last change mentioned also influences other things. For example, “$string .= $thing” has become “$string ~= $thing” for concatenation.

What Sucks about Perl 6

Oh my god everything! (I thought at first). I really did. And I kept running into problems and errors that naturally were due to Perl6’s newness breaking it, and not my own ignorance…. but of course, like most ignorance, it doesn’t recognize itself — and it turns out that every time I thought I had found a bug in Perl6, it was really me not understanding how something had changed, and why.

I may run into legitimate bugs at some point though! I’m just now starting to really dive in. But they’ve had a very long time to work through things.

But it does suck that there are odd little differences between Perl 5 and Perl 6 here and there that can cause problems. But that’s the nature of change. So far nothing sucks really, once I move aside my own irritation at having to learn a few new things.

What’s Surprising about Perl 6

What surprises me most about Perl 6 is that Larry and the team have managed to preserve so much of what was truly monumentally great about Perl 5 and bring that into a context that is offering so many conveniences that were previously only attainable by using sometimes kludgey modules.  For example, I love how we handle command line arguments now. I’ll get into that. No more real need for stuff like Getopt::Long for most things.

Also I had heard that Perl 6 takes a long time to start up and actually run something. This is not the case, at least any more. It feels pretty instantaneous so far, and considering that it has the functionality of Moose built in, that’s awesome! 🙂

I haven’t done any performance metrics on it, and don’t really plan to.  Well, maybe. But it has been a lot more fun than I imagined, and I’m actually looking forward to stopping writing this and getting back to more using it.

Another thing that is surprising to me is its handling of numerics — the math is damn good, and you don’t get weird rounding errors from what I’ve seen. Also it has these magical ranges of numbers that can go clear up to infinity, or skip in defined sequences, and has even type testing built in like is this number a prime… ?

What’s surprising is that it feels like it’s a whole magical world again, and one where you can use convention, or establish your own conventions just as easily.

Anyway, I’ll leave this at that. I’ve really only taken the first little test drive, and not put it through an real industrial rigors. That will be coming as I gain deeper familiarity with Perl 6 — which I have decided now, I intend to do.

Perl Major Mode for Emacs – How to Install

If you are an emacs user and the normal Perl mode is not formatting your newfangled Perl6 code properly, you may need a different mode than perl-mode.

Thanks to Hinrik Örn Sigurðsson you can get an Emacs major mode called perl6-mode from Github. And if you’re lazy like me, you’ll probably just want to have Emacs install and compile it for you from the MELPA archive.

To do that, in your .emacs file put:

(require 'package)
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))

This will allow you to download emacs packages/modes from the MELPA archive for Emacs. Then in Emacs do a:

M-x package-refresh-contents
M-x package-install
perl6-mode

From that point on, emacs should recognize Perl6 by file extensions or even if you have the magic headers like #!/usr/bin/env perl6 at the top of your programs.

So far, I’ve found Hinrik’s perl6-mode to be a godsend — I was going nuts living with the perl-mode getting confused. The only thing I’ve noticed is that it doesn’t colorize pod stuff the way I’m used to.

Thank you Hinrik!!!

Fixing the Reckless Debian Security Update to Samba 4.2

If you’re like me, and were wanting to get the Samba security fixes in place quickly for Debian Jessie version of SAMBA — you’ll maybe have learned that they decided not to backport the fixes for man-in-the-middle, but instead upgrade the whole thing to SAMBA 4.2.

That’s all well and good — but the security update broke SAMBA on my Debian boxes in the test lab (which run Jessie).

The fix was pretty simple after some fairly serious wasted time Googling for it. By default, it SAMBA will now require the winbind package to be installed, otherwise it will silently fail to start.

The symptom I had was that internal DNS resolution for SAMBA stopped working.  And nothing was bound to port 53.

Of course, that was because Samba wasn’t starting at all.

I hope that helps someone — just make sure winbind is installed and running.

They really should have tested this better, even with such a bad flaw in the Microsoft and Samba stuff.

Pipe to Perl for the Quick and Dirty Big Jobs

I always find myself using Perl for systems administration tasks, particularly when you must do things on many systems at once. It is in Perl’s DNA to be down in the trenches of the operating system.

For example, last night I had an NTP configuration file I needed to get out to a few dozen machines. I had a list of them from a display of managed machines, and they were tabulated like this:

62 comp1.dom.com 8 Workstation 6 Company, Inc.
61 comp2.dom.com 8 Workstation 6 Company, Inc.
63 comp3.dom.com 8 Workstation 6 Company, Inc.

And I need to get an NTP configuration file to each. Sure, you can write a shell script using sed to do it just as easy, but with Perl you do get a little more flexibility to do the strange things that must be done from time to time.

#!/usr/bin/env perl

foreach (<STDIN>) {
    my($id, $host, $ignore) = split;
    print `scp -v ntp.conf root\@$host:/etc`;
}

Of course, if you like to live dangerously, you can always put in full ssh commands. You can make the command you want to run be an argument. You can use any of the numerous Perl modules available to handle specific things or concerns.

The command for this is something like:

list_hosts | my_perl_program

Perl just lets you read from STDIN as a normal pipe, with nothing extra you have to do. Columnar/tabular data – it’s so easy. Calling a shell command and sending the output anywhere? Right there.

Edit that to reload your NTP daemon on each. Or make the command an argument when you call the program. Or an option, using Getopt::Long, or whatever you like.

Anyway, my suggestion here, start using some Perl. For admin purposes, it’s like the shell, on multi-dimensional steroids that don’t cause cancer.