I have an S3 bucket that contains several hundred files in a folder. I needed to copy those files into a different folder in another bucket. Sounds simple enough? but was unable to find a simple way to do this through the AWS Console. I found a number of stack overflow articles that talked about using Sync, or downloading the files and re-uploading them. None of which sounded particularly appealing.
In the end I just wrote this bash one liner (which I can probably optimise further by not repeating the sourcebucket / sourcefolder three times):
This just uses s3cmd to list all the files in the bucket/folder I wish to copy from. The output of that is piped to awk which I use to extract the s3 url of each file. I then use tail to remove the first line which I don’t need. I then use sed to build up a ‘s3cmd cp’ command which copies the file from its original location to my new location.
If anyone can suggest a better way that doesnt require me having to download the source files … I’d love to hear it.
If you can’t see the embedded Gist above then you can view it here.
Daniel Suarez talk is one I think everyone should watch. The more I consider his words the more I’m convinced that he is right in calling for international ban on the development and deployment of autonomous killer robots. He makes many good points during the talk but here are the ones that really made me stop and think:
because as we migrate lethal decision-making from humans to software, we risk not only taking the humanity out of war, but also changing our social landscape entirely, far from the battlefield. That’s because the way humans resolve conflict shapes our social landscape … Now if responsibility and transparency are two of the cornerstones of representative government, autonomous robotic weapons could undermine both … And this is why we need an international treaty on robotic weapons, and in particular a global ban on the development and deployment of killer robots. Now we already have international treaties on nuclear and biological weapons, and, while imperfect, these have largely worked. But robotic weapons might be every bit as dangerous, because they will almost certainly be used, and they would also be corrosive to our democratic institutions.
I’ve been reading a lot over the last few months, and whilst I wish I had the time to write a fuller review of each of the following, I know I just wont have the time. Therefore I’ve written a short review for each of these titles.
On Internet Freedom – Marvin Ammori Marvin Ammori is a leading campaigner and legal expert on net neutrality and keeping the internet free. This is an excellent book – and it brings together a set of stories that underline why the Internet changes not only how we think about free speech, but how we must seek to protect and promote it. The message is a simple one – that the Internet has come to be the most important engine of free expression in history. Yet it could also be broken due to the missuse of government and corporate power – motivated by fear, greed, and misguided notions of responsibility. This book does succeed in providing some hope that the spirit of activism on the Net is mobilising people to defend their rights.This book was written against the backdrop of SOPA and was therefore very timely.
Mortality – Christopher Hitchens I’ve always admired Christopher Hitchens, and over the years have read many of his articles and books. He was never one to shy away from making a principled argument. It comes as no surprise that one of the most remarkable polemicists this country has ever produced didn’t leave without having a few important things to say. It is a sobering and often harrowing account of Hitchens final “year of living dyingly” as describes his battle with Cancer in seven essays. The essays begin with Hitchens being diagnosed in June 2010. The openness with which he relates the news is, I think, brave, and his shock is palpable. But I think what I admired the most was rather succumb to rage he instead favours Curiosity. The cancer robbed him of his two main attributes: his voice and the energy to write and its his reflections on these two aspects of his illness that are the most poignant.
Church of Fear : Inside the weird world of Scientology – John Sweeney There is something fascinating about cults and I’ve read about many over the years in particular learning about the human rights abuses they were guilty of, and often trying to understand how they got away with it. Scientology is one of the worst offenders. I was introduced to it when, at the tender age of 17, someone tried to sell my a copy of Dianetics after asking me to take a “test”, I had time to kill so humoured the individual but I recall after a 20 minute interview I’d learned enough to know it wasn’t something I wanted to be part of. Many people have seen John Sweeney’ documentaries about the Church of Scientology which were disturbing and often difficult to watch. This book lays bare just how terrifying this organisation has become. Sweeney details many examples of people who have either left the church and are critical of it and then extraordinary lengths to which the Church will go to harass and besmirch these individuals. Found myself agreeing with Sweeney when he wrote:
It is as if there is in the United States an eleventh commandment: “Thou shalt not criticize another man’s religion” The danger is that in America they are so afraid of religious un-freedom that they fear to discriminate between a religion and a confidence trick.
How they started in tough times – David LesterI’ve never read any of the other books in the “How they started” series. However this one was recommended to me by a friend, and it was well worth reading. The book profiles a wide variety of businesses most you will have heard of. Many insights are given into just how these businesses got started and survived in hard economic times. Given the current economic climate, I found the book actually offers hope rather than the typical doom and gloom. The particular companies that are profiled include Wikipedia, Moonpig, Mumsnet, LinkedIn, Walt Disney, Penguin and many more. Pretty inspirational actually.
After the Apocalypse – Maureen F. McHugh I love reading science fiction and fantasy, this was a wonderful collection of short stories that are about life after an Apocalypse. Any large enough catastrophe is an apocalypse of sorts, leaving lives altered in its wake, with survivors who still need to live in a changed world. What I really liked about this collection is that those survivors are simply everyday people caught up in events, and the choices they make are as varied as human beings can be. With “near-future” being one of the hot topics in science fiction at the moment After the Apocalypse succeeds where others don’t by depicting ordinary people trying to get on with their ordinary lives as best they can, despite the hopelessness and horror around them.
Tomorrow is the fifth anniversary of my father’s death, it’s hard to believe that its been five years, and yet here we are; so much has happened, so much has changed and yet so much remains the same. I miss him so very much, I miss his wisdom, his humour, his strength but most of all I miss the comfort I felt simply knowing he was there. It was while I was thinking of him that I recalled this poem by Henry Wadsworth Longfellow … which somehow feels apt…
(What the heart of the young man said to the psalmist)
Tell me not, in mournful numbers,
"Life is but an empty dream!"
For the soul is dead that slumbers,
And things are not what they seem.
Life is real! Life is earnest!
And the grave is not its goal;
"Dust thou art, to dust returnest,"
Was not spoken of the soul.
Not enjoyment, and not sorrow,
Is our destined end or way;
But to act, that each tomorrow
Finds us farther than today.
Art is long, and Time is fleeting,
And our hearts, though stout and brave,
Still, like muffled drums, are beating
Funeral marches to the grave.
In the world's broad field of battle,
In the bivouac of Life,
Be not like dumb, driven cattle!
Be a hero in the strife!
Trust no Future, howe'er pleasant!
Let the dead Past bury its dead!
Act, -act in the living Present!
Heart within, and God o'erhead!
Lives of great men all remind us
We can make our lives sublime,
And, departing, leave behind us
Footprints on the sands of time;
Footprints, that perhaps another,
Sailing o'er life's solemn main,
A forlorn and shipwrecked brother,
Seeing, shall take heart again.
Let us, then, be up and doing,
With a heart for any fate;
Still achieving, still pursuing,
Learn to labour and to wait.
I’ve been working more and more with nodejs and have to say I am really loving how easy it’s been to get to grips with. I’ll be posting up more about how I’m using node and the problems I’m using it to solve over the coming weeks. However I wanted to illustrate just how simple it is do something that would be more work in other languages such as PHP.
So here’s a quick example I created that illustrates how to make a simple JSONP call to a nodejs + express server.
Here’s the server side code:
and here’s the client side Html code:
To enable jsonp callback support in an ExpressJS application you just have to include the line:
If entrepreneurship is a battle, most casualties stem from friendly fire or self inflicted wounds
I’ve really enjoyed reading Noam Wasserman’s ‘The Founders Dilemma: Anticipating and Avoiding the Pitfalls That Can Sink a Startup‘. This is possibly best described as book of research and case studies conducted into starting up a company from the perspective of an entrepreneur. Wasserman focuses on the key dilemmas any founder of a startup will face; these include career dilemmas, dilemmas with co-founders, acquisitions, investor dilemmas, recruitment, exit dilemmas and many others. The book analyses, quantitatively, the early decisions by entrepreneurs that can make or break a startup; what he has done here is conducted real research to get beyond anecdotes and case studies to more general patterns.
What occurred to me was that many of these dilemmas apply to any situation where a leader might be faced with building a team, finding the right people to embark on a journey together in order to achieve a shared goal. What makes this book such compelling reading are the real world examples that Wasserman cites, and the candor with which he analyses the decisions that were made and describes the real effects and consequences of those decisions.
The original film is set in 17th-century Japan, where an era of peace causes the Shogunate to breakup the various warrior clans, throwing thousands of samurai out of work and into poverty. Such a fate is abhorrent to a Samurai, and many prefer ritual suicide (hara kiri) than to live their lives destitute.
The film tells the story of an old warrior, Hanshiro Tsugumo (Tatsuya Nakadai), who arrives seeking admittance to the house of a feudal lord in order to commit ritual suicide.
The reason behind Hanshiro’ arrival is far more complex, as his real motive is to exact revenge against the house. Whilst preparing to commit the act, Hanshiro recounts, in flashback, to the various assembled members of the house, the tragic story of his son-in-law who was forced to sell his real sword to support his sick wife and child. The story comes to a head when it is revealed his son in law was forced to commit ritual suicide with a dull bamboo blade when he came to the same house seeking work.
The original film was brilliant, and I’m really excited to see how Miike has re-imagined the story. There’s a trailer here.
“Lack of passion leads to poor performance, which will damage any future chance of success. As a consequence it becomes progressively more and more difficult to be passionate”; Inertia is a self fulfilling prophecy.
It’s been so long since I’ve actually written a blog post I thought I’d try something different. Start with something that sounds like a conclusion … and try to piece it all together.
…but first a personal note …
I haven’t written anything particularly meaningful on this blog in a long time, somewhere along the way I lost my voice. I’ve been trying to understand why; some of it I cant remember or am probably still unwilling to remember at least not in enough detail to examine those feelings honestly. It’s amusing, at least to me, that all of this was brought into sharp relief when I read this simple sentence yesterday on an old friends blog:
Not being able to speak manifests in not being able to write too.
That is so fucking true.
Which in turn linked through to a different post that also had a profound effect on me; by profound I mean made me feel angry and ashamed all at once:
What does it feel like to remain silent when you should have said something? I bet you can think of occasions looking back when you wished you had found the words, any words, to say something. Say Something.
When I first started blogging it was, in no small part, due to a request from another friend, Ian Davis who back in 2006 asked me to start blogging and sharing my thoughts. Whilst I was nervous and unsure of myself, Ian convinced me to try and more importantly – to believe in myself. I was still fairly new to Talis at the time and I remember asking him “what if I say something you don’t like? or the company doesn’t like?” and Ian’s response was something along the lines of “its your blog and your voice … you can say whatever you want”. Of course I realised that he wasn’t giving me free reign to give away our trade secrets (not that we had many), or use it as a platform from which to hurl abuse at managers. What he was doing was saying its ok to question, it ok to share thoughts, even concerns because if we are an organisation that values our people then we have to encourage them to have a voice. For me it was this that made us far more ‘human’ or ‘people’ focussed than the company I left in order to join Talis. Ian left Talis a few months ago, I know he has a lot to deal with, and he is! But I do miss him. He has always inspired me and he still does; for that I will always be grateful to him. I wish I had said that to him more often, I hope its something he already knew.
So, back to Attitude reflecting leadership.
This is a rather old post on The Apathy Cycle vs The Passion Cycle. The quote at the beginning of this post was taken from this. Whilst the post itself is quite short there’s some interesting discussion in the comments that is also worth reflecting on. What do Passion and Apathy have to do with Attitude and Leadership … perhaps nothing … perhaps everything.
There’s been a lot of change recently, not only at Talis but also in other aspects of my life – family, friends, other projects I’m involved in outside of work. About the only thing that seems to remain constant is the fact that things keep changing. Dealing with change is not always easy – But you deal with it, right? I used to believe that everything changes and all that matters is how we, as individuals, choose to deal with it. But I’ve been re-thinking my position on that recently. Particularly when I think about it in the light of those things I’ve been really passionate about recently and those things I’ve been pretty apathetic towards.
The catalyst for this was several closely related questions that I was asked by two different people in two completely unrelated contexts (one was at work and the other on a different local project I volunteer with). But for the purposes of this I’m going to reduce those to just one and use that to frame the rest of this discussion:
What is it that makes the kind of leader you would choose to follow?
I’ve read books on leadership, team building and organisational culture; I’ve discussed these at length with others often more experienced in this subject area than I am. There are so many different ways of answering ‘what is a good leader’, and often when people answer they do so by pointing out the differences between Leaders and Managers. There is an important distinction, but difficult at times to articulate – I think what they all agree on is that the difference lies in the way that Leaders or Managers motivate the people who work or follow them. I know for some that’s a gross over simplification, so I’ll try to qualify this a little more. Leaders have followers, whereas Managers tend to have subordinates. Again, this is also over simplified since in many situations the same individual will have a Leadership position that requires him/her to Manage others. They are not mutually exclusive.
For me the leadership qualities that are required to make a good leader will vary in different companies, teams and situations. They are entirely context-dependent. What does that mean? every situation we face is different – I am a leader to some but a follower of others. No one is always just one or the other. One of the best examples of this, certainly one that helped me understand the importance of context was the play The Admirable Crichton, in which a Lord and his Butler swap their roles as leader and servant, as the situation changes. For example: when on a desert island the butler’s practical skills are essential for survival.
I might argue that in general a good leader is someone who thinks strategically, has a vision that is the source of their passion and communicates it effectively to others, inspiring them to follow; in other words enthusing others to work towards that vision because they too believe in it. Good leaders are also the ones that understand thats it’s important to boost the self esteem of others, it’s amazing what people can achieve when they believe in themselves. If you want to be technical then yes I am listing some of the qualities that are attributed to transformational leadership as opposed to transactional leadership where people are often motivated by reward or punishment and there’s a clear chain of command. I found the table at the end of this page helped as a talking point during one of the discussions I had. I don’t entirely agree with it but it was useful nonetheless. The other thing that helped was the film Twelve O’Clock High which was a more visual way of examining the effectiveness of different leadership styles and in different contexts! (had the damnedest time convincing them to watch a black and white film).
I personally believe that certainly with relation to the work that I do, or the projects I’m involved in, particularly those in and around social innovation, tapping into people’s passions and empowering them works better than diktat or command and control – I’ve witnessed for example at P2PU how far passion can take a community, that is led in an open and transparent way towards shared vision. I’ve also seen how corrosive it is when people’s motivations aren’t aligned behind a shared vision. It creates uncertainty, disillusionment and possibly most damning of all … a fundamental lack of trust. Often actually more through a lack of communication and transparency than anything else.
So when the question above was put me most recently, I answered:
‘the leader provides a vision, but he cant get there on his/her own – First I ask is the vision something that I believe in, does it inspire me, is it something I feel passionate about;. BUT then I ask myself how is that shared vision reflected in everyone else; In their words and actions … because you aren’t just following a person your also joining a team/community/movement.’.
The prevailing attitudes embodied within any team/community/movement are a reflection of its leadership. Apathy isn’t always a result of bad members, more often its a result of uninspired leadership.
That’s probably enough to think about for now. Hope this has been useful for those who asked.
It’s been a while since I read something that moved me as much as this short essay by an 11th Grader.
2012 Martin Luther King, Jr. Writing Awards
Prose: High School
Fighting a Forbidden Battle: How I Stopped Covering Up for a Hidden Wrong
11th grade, Winchester Thurston
I once belonged to a wonderful religion. I belonged to a religion that allows those of us who believe in it to feel that we are the greatest people in the world—and feel sorry for ourselves at the same time. Once, I thought that I truly belonged in this world of security, self-pity, self-proclaimed intelligence, and perfect moral aesthetic. I thought myself to be somewhat privileged early on. It was soon revealed to me, however, that my fellow believers and I were not part of anything so flattering.
Although I was fortunate enough to have parents who did not try to force me into any one set of beliefs, being Jewish was in no way possible to escape growing up. It was constantly reinforced at every holiday, every service, and every encounter with the rest of my relatives. I was forever reminded how intelligent my family was, how important it was to remember where we had come from, and to be proud of all the suffering our people had overcome in order to finally achieve their dream in the perfect society of Israel.
This last mandatory belief was one which I never fully understood, but I always kept the doubts I had about Israel’s spotless reputation to the back of my mind. “Our people” were fighting a war, one I did not fully comprehend, but I naturally assumed that it must be justified. We would never be so amoral as to fight an unjust war. Yet as I came to learn more about our so-called “conflict” with the Palestinians, I grew more concerned. I routinely heard about unexplained mass killings, attacks on medical bases, and other alarmingly violent actions for which I could see no possible reason. “Genocide” almost seemed the more appropriate term, yet no one I knew would have ever dreamed of portraying the war in that manner; they always described the situation in shockingly neutral terms. Whenever I brought up the subject, I was always given the answer that there were faults on both sides, that no one was really to blame, or simply that it was a “difficult situation.” It was not until eighth grade that I fully understood what I was on the side of. One afternoon, after a fresh round of killings was announced on our bus ride home, I asked two of my friends who actively supported Israel what they thought. “We need to defend our race,” they told me. “It’s our right.”
“We need to defend our race.”
Where had I heard that before? Wasn’t it the same excuse our own country had used to justify its abuses of African-Americans sixty years ago? In that moment, I realized how similar the two struggles were—like the white radicals of that era, we controlled the lives of another people whom we abused daily, and no one could speak out against us. It was too politically incorrect to do so. We had suffered too much, endured too many hardships, and overcome too many losses to be criticized. I realized then that I was in no way part of a “conflict”—the term “Israeli/Palestinian Conflict” was no more accurate than calling the Civil Rights Movement the “Caucasian/ African-American Conflict.” In both cases, the expression was a blatant euphemism: it gave the impression that this was a dispute among equals and that both held an equal share of the blame. However, in both, there was clearly an oppressor and an oppressed, and I felt horrified at the realization that I was by nature on the side of the oppressors. I was grouped with the racial supremacists. I was part of a group that killed while praising its own intelligence and reason. I was part of a delusion.
I thought of the leader of the other oppressed side of years ago, Martin Luther King. He too had been part of a struggle that had been hidden and glossed over for the convenience of those against whom he fought. What would his reaction have been? As it turned out, it was precisely the same as mine. As he wrote in his letter from Birmingham Jail, he believed the greatest enemy of his cause to be “Not the White Citizen’s Counciler or the Ku Klux Klanner, but the white moderate, who…lives by a mythical concept of time…. Lukewarm acceptance is much more bewildering than outright rejection.” When I first read those words, I felt as if I were staring at myself in a mirror. All my life I had been conditioned to simply treat the so-called conflict with the same apathy which King had so forcefully condemned. I, too, held the role of an accepting moderate. I, too, “lived by a mythical concept of time,” shrouded in my own surreal world and the set of beliefs that had been assigned to me. I had never before felt so trapped.
I decided to make one last appeal to my religion. If it could not answer my misgivings, no one could. The next time I attended a service, there was an open question-and-answer session about any point of our religion. I wanted to place my dilemma in as clear and simple terms as I knew how. I thought out my exact question over the course of the seventeen-minute cello solo that was routinely played during service. Previously, I had always accepted this solo as just another part of the program, yet now it seemed to capture the whole essence of our religion: intelligent and well-crafted on paper, yet completely oblivious to the outside world (the soloist did not have the faintest idea of how masterfully he was putting us all to sleep). When I was finally given the chance to ask a question, I asked, “I want to support Israel. But how can I when it lets its army commit so many killings?” I was met with a few angry glares from some of the older men, but the rabbi answered me. “It is a terrible thing, isn’t it?” he said. “But there’s nothing we can do. It’s just a fact of life.” I knew, of course, that the war was no simple matter and that we did not by any means commit murder for its own sake, but to portray our thousands of killings as a “fact of life” was simply too much for me to accept. I thanked him and walked out shortly afterward. I never went back. I thought about what I could do. If nothing else, I could at least try to free myself from the burden of being saddled with a belief I could not hold with a clear conscience. I could not live the rest of my life as one of the pathetic moderates whom King had rightfully portrayed as the worst part of the problem. I did not intend to go on being one of the Self-Chosen People, identifying myself as part of a group to which I did not belong.
It was different not being the ideal nice Jewish boy. The difference was subtle, yet by no means unaffecting. Whenever it came to the attention of any of our more religious family friends that I did not share their beliefs, I was met with either a disapproving stare and a quick change of the subject or an alarmed cry of, “What? Doesn’t Israel matter to you?” Relatives talked down to me more afterward, but eventually I stopped noticing the way adults around me perceived me. It was worth it to no longer feel as though I were just another apathetic part of the machine.
I can obviously never know what it must have been like to be an African-American in the 1950s. I do feel, however, as though I know exactly what it must have been like to be white during that time, to live under an aura of moral invincibility, to hold unchallengeable beliefs, and to contrive illusions of superiority to avoid having to face simple everyday truths. That illusion was nice while it lasted, but I decided to pass it up. I have never been happier.
Anyone can achieve their fullest potential,
who we are might be predetermined, but the path
we follow is always of our own choosing. We should
never allow our fears or the expectations of others
to set the frontiers of our destiny. Your destiny can't
be changed but, it can be challenged.
Every man is born as many men and dies as a single one.