Copy files from one S3 bucket to another

I have an S3 bucket that contains several hundred files in a folder. I needed to copy those files into a different folder in another bucket. Sounds simple enough? but was unable to find a simple way to do this through the AWS Console. I found a number of stack overflow articles that talked about using Sync, or downloading the files and re-uploading them. None of which sounded particularly appealing.

In the end I just wrote this bash one liner (which I can probably optimise further by not repeating the sourcebucket / sourcefolder three times):

This just uses s3cmd to list all the files in the bucket/folder I wish to copy from. The output of that is piped to awk which I use to extract the s3 url of each file. I then use tail to remove the first line which I don’t need. I then use sed to build up a ‘s3cmd cp’ command which copies the file from its original location to my new location.

If anyone can suggest a better way that doesnt require me having to download the source files … I’d love to hear it.
If you can’t see the embedded Gist above then you can view it here.

Daniel Suarez on why the kill decision should not belong to a robot

Daniel Suarez talk is one I think everyone should watch. The more I consider his words the more I’m convinced that he is right in calling for international ban on the development and deployment of autonomous killer robots. He makes many good points during the talk but here are the ones that really made me stop and think:

because as we migrate lethal decision-making from humans to software, we risk not only taking the humanity out of war, but also changing our social landscape entirely, far from the battlefield. That’s because the way humans resolve conflict shapes our social landscape … Now if responsibility and transparency are two of the cornerstones of representative government, autonomous robotic weapons could undermine both … And this is why we need an international treaty on robotic weapons, and in particular a global ban on the development and deployment of killer robots. Now we already have international treaties on nuclear and biological weapons, and, while imperfect, these have largely worked. But robotic weapons might be every bit as dangerous, because they will almost certainly be used, and they would also be corrosive to our democratic institutions.

Books, books, books …

I’ve been reading a lot over the last few months, and whilst I wish I had the time to write a fuller review of each of the following, I know I just wont have the time. Therefore I’ve written a short review for each of these titles.

On Internet Freedom – Marvin Ammori
Marvin Ammori is a leading campaigner and legal expert on net neutrality and keeping the internet free. This is an excellent book – and it brings together a set of stories that underline why the Internet changes not only how we think about free speech, but how we must seek to protect and promote it. The message is a simple one – that the Internet has come to be the most important engine of free expression in history. Yet it could also be broken due to the missuse of government and corporate power – motivated by fear, greed, and misguided notions of responsibility. This book does succeed in providing some hope that the spirit of activism on the Net is mobilising people to defend their rights.This book was written against the backdrop of SOPA and was therefore very timely.

Mortality – Christopher Hitchens
I’ve always admired Christopher Hitchens, and over the years have read many of his articles and books. He was never one to shy away from making a principled argument. It comes as no surprise that one of the most remarkable polemicists this country has ever produced didn’t leave without having a few important things to say. It is a sobering and often harrowing account of Hitchens final “year of living dyingly” as describes his battle with Cancer in seven essays. The essays begin with Hitchens being diagnosed in June 2010. The openness with which he relates the news is, I think, brave, and his shock is palpable. But I think what I admired the most was rather succumb to rage he instead favours Curiosity. The cancer robbed him of his two main attributes: his voice and the energy to write and its his reflections on these two aspects of his illness that are the most poignant.

Church of Fear : Inside the weird world of Scientology – John Sweeney
There is something fascinating about cults and I’ve read about many over the years in particular learning about the human rights abuses they were guilty of, and often trying to understand how they got away with it. Scientology is one of the worst offenders. I was introduced to it when, at the tender age of 17, someone tried to sell my a copy of Dianetics after asking me to take a “test”, I had time to kill so humoured the individual but I recall after a 20 minute interview I’d learned enough to know it wasn’t something I wanted to be part of. Many people have seen John Sweeney’ documentaries about the Church of Scientology which were disturbing and often difficult to watch. This book lays bare just how terrifying this organisation has become. Sweeney details many examples of people who have either left the church and are critical of it and then extraordinary lengths to which the Church will go to harass and besmirch these individuals. Found myself agreeing with Sweeney when he wrote:

It is as if there is in the United States an eleventh commandment: “Thou shalt not criticize another man’s religion” The danger is that in America they are so afraid of religious un-freedom that they fear to discriminate between a religion and a confidence trick.

Async Javascript: Recipes for Event Driven Code – Trevor Burnham
If you are interested in learning more about asynchronous programming in JavaScript I highly recommend this book. The first chapter alone on “The JavaScript Event Model” is worth the price of the book. Technical books about programming languages often focus on the mechanics of a particular language – how it works. Occasionally a book (or an article or blog post ) comes along that transforms your entire perception of a language. This, for me, is definitely the latter. This book was in no small part the catalyst for my current love affair with NodeJS. In a very clear and concise manner, concepts such as the Pub/Sub model, custom events, Promises/Deferreds and Web Workers are detailed with good clear code examples. I think its only available on the Kindle now, but if you’re starting off with NodeJS, or your battling with async code read this book!

How they started in tough times – David Lester
I’ve never read any of the other books in the “How they started” series. However this one was recommended to me by a friend, and it was well worth reading. The book profiles a wide variety of businesses most you will have heard of. Many insights are given into just how these businesses got started and survived in hard economic times. Given the current economic climate, I found the book actually offers hope rather than the typical doom and gloom. The particular companies that are profiled include Wikipedia, Moonpig, Mumsnet, LinkedIn, Walt Disney, Penguin and many more. Pretty inspirational actually.


After the Apocalypse – Maureen F. McHugh
I love reading science fiction and fantasy, this was a wonderful collection of short stories that are about life after an Apocalypse. Any large enough catastrophe is an apocalypse of sorts, leaving lives altered in its wake, with survivors who still need to live in a changed world. What I really liked about this collection is that those survivors are simply everyday people caught up in events, and the choices they make are as varied as human beings can be. With “near-future” being one of the hot topics in science fiction at the moment After the Apocalypse succeeds where others don’t by depicting ordinary people trying to get on with their ordinary lives as best they can, despite the hopelessness and horror around them.

A Psalm of Life

Tomorrow is the fifth anniversary of my father’s death, it’s hard to believe that its been five years, and yet here we are; so much has happened, so much has changed and yet so much remains the same. I miss him so very much, I miss his wisdom, his humour, his strength but most of all I miss the comfort I felt simply knowing he was there. It was while I was thinking of him that I recalled this poem by Henry Wadsworth Longfellow … which somehow feels apt…

(What the heart of the young man said to the psalmist)

Tell me not, in mournful numbers,
"Life is but an empty dream!"
For the soul is dead that slumbers,
And things are not what they seem.

Life is real! Life is earnest!
And the grave is not its goal;
"Dust thou art, to dust returnest,"
Was not spoken of the soul.

Not enjoyment, and not sorrow,
Is our destined end or way;
But to act, that each tomorrow
Finds us farther than today.

Art is long, and Time is fleeting,
And our hearts, though stout and brave,
Still, like muffled drums, are beating
Funeral marches to the grave.

In the world's broad field of battle,
In the bivouac of Life,
Be not like dumb, driven cattle!
Be a hero in the strife!

Trust no Future, howe'er pleasant!
Let the dead Past bury its dead!
Act, -act in the living Present!
Heart within, and God o'erhead!

Lives of great men all remind us
We can make our lives sublime,
And, departing, leave behind us
Footprints on the sands of time;

Footprints, that perhaps another,
Sailing o'er life's solemn main,
A forlorn and shipwrecked brother,
Seeing, shall take heart again.

Let us, then, be up and doing,
With a heart for any fate;
Still achieving, still pursuing,
Learn to labour and to wait.

nodejs + express with jsonp example

I’ve been working more and more with nodejs and have to say I am really loving how easy it’s been to get to grips with. I’ll be posting up more about how I’m using node and the problems I’m using it to solve over the coming weeks. However I wanted to illustrate just how simple it is do something that would be more work in other languages such as PHP.

So here’s a quick example I created that illustrates how to make a simple JSONP call to a nodejs + express server.

Here’s the server side code:

and here’s the client side Html code:

To enable jsonp callback support in an ExpressJS application you just have to include the line:

app.enable(“jsonp callback”);

Once you’ve done this you can use the .json() method on the Response object to handle everything for you. So in my example above any HTTP GET request to /foo?cb=myfunction would return myfunction(“hello world”); with the Content-Type header set to text/javascript.

The Founder’s Dilemma

If entrepreneurship is a battle, most casualties stem from friendly fire or self inflicted wounds

I’ve really enjoyed reading Noam Wasserman’s ‘The Founders Dilemma: Anticipating and Avoiding the Pitfalls That Can Sink a Startup‘. This is possibly best described as book of research and case studies conducted into starting up a company from the perspective of an entrepreneur. Wasserman focuses on the key dilemmas any founder of a startup will face; these include career dilemmas, dilemmas with co-founders, acquisitions, investor dilemmas, recruitment, exit dilemmas and many others. The book analyses, quantitatively, the early decisions by entrepreneurs that can make or break a startup; what he has done here is conducted real research to get beyond anecdotes and case studies to more general patterns.

What occurred to me was that many of these dilemmas apply to any situation where a leader might be faced with building a team, finding the right people to embark on a journey together in order to achieve a shared goal. What makes this book such compelling reading are the real world examples that Wasserman cites, and the candor with which he analyses the decisions that were made and describes the real effects and consequences  of those decisions.

Hara-Kiri : Death of a Samurai

HaraKiri Death of a Samurai posterQuite excited, learned recently that Takashi Miike‘s, Hara-Kiri: Death of a Samurai, will be available here in the UK fairly soon. Right now all I know is that it’s an adaptation of Masaki Kobayashi‘s 1962 film Harakiri, which is one of my all time favourites.

The original film is set in 17th-century Japan, where an era of peace causes the Shogunate to breakup the various warrior clans, throwing thousands of samurai out of work and into poverty. Such a fate is abhorrent to a Samurai, and many prefer ritual suicide (hara kiri) than to live their lives destitute.

The film tells the story of an old warrior, Hanshiro Tsugumo (Tatsuya Nakadai), who arrives seeking admittance to the house of a feudal lord in order to commit ritual suicide.

The reason behind Hanshiro’ arrival is far more complex, as his real motive is to exact revenge against the house. Whilst preparing to commit the act, Hanshiro recounts, in flashback, to the various assembled members of the house, the tragic story of his son-in-law  who was forced to sell his real sword to support his sick wife and child. The story comes to a head when it is revealed his son in law was forced to commit ritual suicide with a dull bamboo blade when he came to the same house seeking work.

The original film was brilliant, and I’m really excited to see how Miike has re-imagined the story. There’s a trailer here.

Why attitude reflects leadership

“Lack of passion leads to poor performance, which will damage any future chance of success. As a consequence it becomes progressively more and more difficult to be passionate”; Inertia is a self fulfilling prophecy.

It’s been so long since I’ve actually written a blog post I thought I’d try something different. Start with something that sounds like a conclusion … and try to piece it all together.

…but first a personal note

I haven’t written anything particularly meaningful on this blog in a long time, somewhere along the way I lost my voice. I’ve been trying to understand why; some of it I cant remember or am probably still unwilling to remember at least not in enough detail to examine those feelings honestly. It’s amusing, at least to me, that all of this was brought into sharp relief when I read this simple sentence yesterday on an old friends blog1:

Not being able to speak manifests in not being able to write too.

That is so fucking true.

Which in turn linked through to a different post that also had a profound effect on me; by profound I mean made me feel angry and ashamed all at once2:

What does it feel like to remain silent when you should have said something? I bet you can think of occasions looking back when you wished you had found the words, any words, to say something. Say Something.

When I first started blogging it was, in no small part, due to a request from another friend, Ian Davis who back in 2006 asked me to start blogging and sharing my thoughts. Whilst I was nervous and unsure of myself, Ian convinced me to try and more importantly – to believe in myself. I was still fairly new to Talis at the time and I remember asking him “what if I say something you don’t like? or the company doesn’t like?” and Ian’s response was something along the lines of “its your blog and your voice … you can say whatever you want”. Of course I realised that he wasn’t giving me free reign to give away our trade secrets (not that we had many), or use it as a platform from which to hurl abuse at managers. What he was doing was saying its ok to question, it ok to share thoughts, even concerns because if we are an organisation that values our people then we have to encourage them to have a voice. For me it was this that made us far more ‘human’ or ‘people’ focussed than the company I left in order to join Talis. Ian left Talis a few months ago, I know he has a lot to deal with, and he is! But I do miss him. He has always inspired me and he still does; for that I will always be grateful to him. I wish I had said that to him more often, I hope its something he already knew.

So, back to Attitude reflecting leadership.

This is a rather old post on The Apathy Cycle vs The Passion Cycle. The quote at the beginning of this post was taken from this. Whilst the post itself is quite short there’s some interesting discussion in the comments that is also worth reflecting on. What do Passion and Apathy have to do with Attitude and Leadership … perhaps nothing … perhaps everything.

There’s been a lot of change recently, not only at Talis but also in other aspects of my life – family, friends, other projects I’m involved in outside of work. About the only thing that seems to remain constant is the fact that things keep changing. Dealing with change is not always easy – But you deal with it, right? I used to believe that everything changes and all that matters is how we, as individuals, choose to deal with it. But I’ve been re-thinking my position on that recently. Particularly when I think about it in the light of those things I’ve been really passionate about recently and those things I’ve been pretty apathetic towards.

The catalyst for this was several closely related questions that I was asked by two different people in two completely unrelated contexts (one was at work and the other on a different local project I volunteer with). But for the purposes of this I’m going to reduce those to just one and use that to frame the rest of this discussion:

What is it that makes the kind of leader you would choose to follow?

I’ve read books on leadership, team building and organisational culture; I’ve discussed these at length with others often more experienced in this subject area than I am. There are so many different ways of answering ‘what is a good leader’, and often when people answer they do so by pointing out the differences between Leaders and Managers. There is an important distinction, but difficult at times to articulate – I think what they all agree on is that the difference lies in the way that Leaders or Managers motivate the people who work or follow them. I know for some that’s a gross over simplification, so I’ll try to qualify this a little more. Leaders have followers, whereas Managers tend to have subordinates. Again, this is also over simplified since in many situations the same individual will have a Leadership position that requires him/her to Manage others. They are not mutually exclusive.

For me the leadership qualities that are required to make a good leader will vary in different companies, teams and situations. They are entirely context-dependent. What does that mean? every situation we face is different – I am a leader to some but a follower of others. No one is always just one or the other. One of the best examples of this, certainly one that helped me understand the importance of context was the play The Admirable Crichton, in which a Lord and his Butler swap their roles as leader and servant, as the situation changes. For example: when on a desert island the butler’s practical skills are essential for survival.

I might argue that in general a good leader is someone who thinks strategically, has a vision that is the source of their passion and communicates it effectively to others, inspiring them to follow; in other words enthusing others to work towards that vision because they too believe in it. Good leaders are also the ones that understand thats it’s important to boost the self esteem of others, it’s amazing what people can achieve when they believe in themselves. If you want to be technical then yes I am listing some of the qualities that are attributed to transformational leadership as opposed to transactional leadership where people are often motivated by reward or punishment and there’s a clear chain of command. I found the table at the end of this page helped as a talking point during one of the discussions I had. I don’t entirely agree with it but it was useful nonetheless. The other thing that helped was the film Twelve O’Clock High which was a more visual way of examining the effectiveness of different leadership styles and in different contexts! (had the damnedest time convincing them to watch a black and white film).

I personally believe that certainly with relation to the work that I do, or the projects I’m involved in, particularly those in and around social innovation, tapping into people’s passions and empowering them works better than diktat or command and control – I’ve witnessed for example at P2PU how far passion can take a community, that is led in an open and transparent way towards shared vision. I’ve also seen how corrosive it is when people’s motivations aren’t aligned behind a shared vision. It creates uncertainty, disillusionment and possibly most damning of all … a fundamental lack of trust. Often actually more through a lack of communication and transparency than anything else.

So when the question above was put me most recently, I answered:

‘the leader provides a vision, but he cant get there on his/her own – First I ask is the vision something that I believe in, does it inspire me, is it something I feel passionate about;. BUT then I ask myself how is that shared vision reflected in everyone else; In their words and actions … because you aren’t just following a person your also joining a team/community/movement.’.

The prevailing attitudes embodied within any team/community/movement are a reflection of its leadership. Apathy isn’t always a result of bad members, more often its a result of uninspired leadership.

That’s probably enough to think about for now. Hope this has been useful for those who asked.

  1. Link,What does absence tell you?[back]
  2. Link, Why your insignificance matters, speak up, act[back]