Merge remote-tracking branch 'upstream/master' into prestine

.github/CONTRIBUTING.md

@@ -48,45 +48,42 @@ can quickly merge or address your contributions.
 
 5. The issue is closed.
 
-## Setting up Go to work on Packer
+## Setting up Go
 
-If you have never worked with Go before, you will have to complete the following
+If you have never worked with Go before, you will have to install its
-steps in order to be able to compile and test Packer. These instructions target
+runtime in order to build packer.
+
+1. [Install go](https://golang.org/doc/install#install)
+
+## Setting up Packer for dev
+
+If/when you have go installed you can already `go get` packer and `make` in
+order to compile and test Packer. These instructions target
 POSIX-like environments (Mac OS X, Linux, Cygwin, etc.) so you may need to
 adjust them for Windows or other shells.
+The instructions below are for go 1.7. or later.
 
-1. [Download](https://golang.org/dl) and install Go. The instructions below are
-   for go 1.7. Earlier versions of Go are no longer supported.
 
-2. Set and export the `GOPATH` environment variable and update your `PATH`. For
+1. Download the Packer source (and its dependencies) by running
-   example, you can add the following to your `.bash_profile` (or comparable
-   shell startup scripts):
-
-```
-export GOPATH=$HOME/go
-export PATH=$PATH:$GOPATH/bin
-```
-
-3. Download the Packer source (and its dependencies) by running
    `go get github.com/hashicorp/packer`. This will download the Packer source to
    `$GOPATH/src/github.com/hashicorp/packer`.
 
-4. When working on Packer, first `cd $GOPATH/src/github.com/hashicorp/packer`
+2. When working on Packer, first `cd $GOPATH/src/github.com/hashicorp/packer`
    so you can run `make` and easily access other files. Run `make help` to get
    information about make targets.
 
-5. Make your changes to the Packer source. You can run `make` in
+3. Make your changes to the Packer source. You can run `make` in
    `$GOPATH/src/github.com/hashicorp/packer` to run tests and build the Packer
    binary. Any compilation errors will be shown when the binaries are
    rebuilding. If you don't have `make` you can simply run
    `go build -o bin/packer .` from the project root.
 
-6. After running building Packer successfully, use
+4. After running building Packer successfully, use
    `$GOPATH/src/github.com/hashicorp/packer/bin/packer` to build a machine and
    verify your changes work. For instance:
    `$GOPATH/src/github.com/hashicorp/packer/bin/packer build template.json`.
 
-7. If everything works well and the tests pass, run `go fmt` on your code before
+5. If everything works well and the tests pass, run `go fmt` on your code before
    submitting a pull-request.
 
 ### Opening an Pull Request

.travis.yml

@@ -6,10 +6,9 @@ sudo: false
 language: go
 
 go:
-  - 1.8.x
   - 1.9.x
-  - 1.x
+  - 1.10.x
-
+  - master
 
 install:
   - make deps
@@ -22,4 +21,6 @@ branches:
     - master
 
 matrix:
+  allow_failures:
+    - go: master
   fast_finish: true

CHANGELOG.md

@@ -1,3 +1,102 @@
+## 1.2.6 (Unreleased)
+
+### IMPROVEMENTS:
+
+* builder/amazon-chroot: New feature `root_volume_tags` to tag the created
+    volumes. [GH-6504]
+* builder/azure: Implement clean_image_name template engine. [GH-6558]
+* builder/digitalocean: Add support for tagging to instances [GH-6546]
+* builder/lxc: Allow unplivileged LXC containers. [GH-6279]
+* builder/oci: Add `metadata` feature to Packer config. [GH-6498]
+* builder/openstack: Add support for ports. [GH-6570]
+* builder/openstack: Add support for getting config from clouds-public.yaml. [GH-6595]
+* builder/openstack: Support Block Storage volumes as boot volume. [GH-6596]
+* builder/openstack: Migrate floating IP usage to Network v2 API from Compute API. [GH-6373]
+* builder/qemu: add ssh agent support. [GH-6541]
+* builder/qemu: New `use_backing_file` feature [GH-6249]
+* builder/vmware-iso: Try to use ISO files uploaded to the datastore when
+    building remotely instead of uploading them freshly every time [GH-5165]
+* command/validate: Warn users if config needs fixing. [GH-6423]
+* post-processor/vagrant: Support for Docker images. [GH-6494]
+* postprocessor/vagrant: Add support for Azure. [GH-6576]
+* provisioner/ansible: Enable {{.WinRMPassword}} template engine. [GH-6450]
+* provisioner/shell-local: Create PACKER_HTTP_ADDR environment variable
+    [GH-6503]
+
+### BUG FIXES:
+* builder/amazon-ebssurrogate: Clean up volumes at end of build. [GH-6514]
+* builder/azure: Generated password satisfies Azure password requirements
+    [GH-6480]
+* builder/lxc: Correctly pass "config" option to "lxc launch". [GH-6563]
+* builder/vmware-iso: Fix crash caused by invalid datacenter url. [GH-6529]
+* core: Better error handling in downloader when connection error occurs.
+    [GH-6557]
+* core: Fix broken pathing checks in checksum files. [GH-6525]
+
+### BACKWARDS INCOMPATIBILITIES:
+* builder/amazon: "owners" field on source_ami_filter is now required for
+    secuirty reasons. [GH-6585]
+
+## 1.2.5 (July 16, 2018)
+
+### BUG FIXES:
+* builder/alickoud: Fix issue where internet_max_bandwidth_out template option
+    was not being passed to builder. [GH-6416]
+* builder/alicloud: Fix an issue with VPC cleanup. [GH-6418]
+* builder/amazon-chroot: Fix communicator bug that broke chroot builds.
+    [GH-6363]
+* builder/amazon: Replace packer's waiters with those from the AWS sdk, solving
+    several timeout bugs. [GH-6332]
+* builder/azure: update azure-sdk-for-go, fixing 32-bit build errors. [GH-6479]
+* builder/azure: update the max length of managed_image_resource_group to match
+    new increased length of 90 characters. [GH-6477]
+* builder/hyper-v: Fix secure boot template feature so that it properly passes
+    the temolate for MicrosoftUEFICertificateAuthority. [GH-6415]
+* builder/hyperv: Fix bug in HyperV IP lookups that was causing breakages in
+    FreeBSD/OpenBSD builds. [GH-6416]
+* builder/qemu: Fix error race condition in qemu builder that caused convert to
+    fail on ubuntu 18.x [GH-6437]
+* builder/qemu: vnc_bind_address was not being passed to qemu. [GH-6467]
+* builder/virtualbox: Allow iso_url to be a symlink. [GH-6370]
+* builder/vmware: Don't fail on DHCP lease files that cannot be read, fixing
+    bug where builder failed on NAT networks that don't serve DHCP. [GH-6415]
+* builder/vmware: Fix bug where we couldn't discover IP if vm_name differed
+    from the vmx displayName. [GH-6448]
+* builder/vmware: Fix validation to prevent hang when remopte_password is not
+    sent but vmware is building on esxi. [GH-6424]
+* builder/vmware:Correctly default the vm export format to ovf; this is what
+    the docs claimed we already did, but we didn't. [GH-4538]
+* communicator/winrm: Revert an attempt to determine whether remote upload
+    destinations were files or directories, as this broke uploads on systems
+    without Powershell installed. [GH-6481]
+* core: Fix bug in parsing of iso checksum files that arose when setting
+    iso_url to a relative filepath. [GH-6488]
+* core: Fix Packer crash caused by improper error handling in the downloader.
+    [GH-6381]
+* fix: Fix bug where fixer for ssh_private_ip that failed when boolean values
+    are passed as strings. [GH-6458]
+* provisioner/powershell: Make upload of powershell variables retryable, in
+    case of system restarts. [GH-6388]
+
+### IMPROVEMENTS:
+* builder/amazon: Add the ap-northeast-3 region. [GH-6385]
+* builder/amazon: Spot requests may now have tags applied using the `spot_tags`
+    option [GH-5452]
+* builder/cloudstack: Add support for Projectid and new config option
+    prevent_firewall_changes. [GH-6487]
+* builder/openstack: Add support for token authorization and cloud.yaml.
+    [GH-6362]
+* builder/oracle-oci: Add new "instance_name" template option. [GH-6408]
+* builder/scaleway: Add new "bootscript" parameter, allowing the user to not
+    use the default local bootscript [GH-6439]
+* builder/vmware: Add support for linked clones to vmware-vmx. [GH-6394]
+* debug: The -debug flag will now cause Packer to pause between provisioner
+    scripts in addition to Packer steps. [GH-4663]
+* post-processor/googlecompute-import: Added new googlecompute-import post-
+    processor [GH-6451]
+* provisioner/ansible: Add new "playbook_files" option to execute multiple
+    playbooks within one provisioner call. [GH-5086]
+
 ## 1.2.4 (May 29, 2018)
 
 ### BUG FIXES:
@@ -52,6 +151,8 @@
 * provisoner/shell-local: New options have been added to create feature parity
     with the shell-local post-processor. This feature now works on Windows
     hosts. [GH-5956]
+* builder/virtualbox: Use HTTPS to download guest editions, now that it's
+    available. [GH-6406]
 
 ## 1.2.3 (April 25, 2018)
 

Makefile

@@ -11,6 +11,8 @@ GOPATH=$(shell go env GOPATH)
 # gofmt
 UNFORMATTED_FILES=$(shell find . -not -path "./vendor/*" -name "*.go" | xargs gofmt -s -l)
 
+EXECUTABLE_FILES=$(shell find . -type f -perm +111 | egrep -v '^\./(vendor/|\.git|bin/|scripts/|pkg/)' | egrep -v '.*(\.sh|\.bats)' | egrep -v './provisioner/ansible/test-fixtures/exit1')
+
 # Get the git commit
 GIT_DIRTY=$(shell test -n "`git status --porcelain`" && echo "+CHANGES" || true)
 GIT_COMMIT=$(shell git rev-parse --short HEAD)
@@ -43,9 +45,11 @@ package:
 	@sh -c "$(CURDIR)/scripts/dist.sh $(VERSION)"
 
 deps:
+	@go get golang.org/x/tools/cmd/goimports
 	@go get golang.org/x/tools/cmd/stringer
 	@go get -u github.com/mna/pigeon
 	@go get github.com/kardianos/govendor
+	@go get golang.org/x/tools/cmd/goimports
 	@govendor sync
 
 dev: deps ## Build and install a development build
@@ -60,7 +64,7 @@ dev: deps ## Build and install a development build
 	@cp $(GOPATH)/bin/packer pkg/$(GOOS)_$(GOARCH)
 
 fmt: ## Format Go code
-	@gofmt -w -s $(UNFORMATTED_FILES)
+	@gofmt -w -s main.go $(UNFORMATTED_FILES)
 
 fmt-check: ## Check go code formatting
 	@echo "==> Checking that code complies with gofmt requirements..."
@@ -73,6 +77,15 @@ fmt-check: ## Check go code formatting
 		echo "Check passed."; \
 	fi
 
+mode-check: ## Check that only certain files are executable
+	@echo "==> Checking that only certain files are executable..."
+	@if [ ! -z "$(EXECUTABLE_FILES)" ]; then \
+		echo "These files should not be executable or they must be white listed in the Makefile:"; \
+		echo "$(EXECUTABLE_FILES)" | xargs -n1; \
+		exit 1; \
+	else \
+		echo "Check passed."; \
+	fi
 fmt-docs:
 	@find ./website/source/docs -name "*.md" -exec pandoc --wrap auto --columns 79 --atx-headers -s -f "markdown_github+yaml_metadata_block" -t "markdown_github+yaml_metadata_block" {} -o {} \;
 
@@ -88,7 +101,7 @@ generate: deps ## Generate dynamically generated code
 	goimports -w common/bootcommand/boot_command.go
 	gofmt -w command/plugin.go
 
-test: deps fmt-check ## Run unit tests
+test: deps fmt-check mode-check ## Run unit tests
 	@go test $(TEST) $(TESTARGS) -timeout=2m
 	@go tool vet $(VET)  ; if [ $$? -eq 1 ]; then \
 		echo "ERROR: Vet found problems in the code."; \

Vagrantfile

@@ -5,6 +5,10 @@ LINUX_BASE_BOX = "bento/ubuntu-16.04"
 FREEBSD_BASE_BOX = "jen20/FreeBSD-12.0-CURRENT"
 
 Vagrant.configure(2) do |config|
+        if Vagrant.has_plugin?("vagrant-cachier")
+                config.cache.scope = :box
+        end
+
 	# Compilation and development boxes
 	config.vm.define "linux", autostart: true, primary: true do |vmCfg|
 		vmCfg.vm.box = LINUX_BASE_BOX

builder/alicloud/ecs/builder.go

@@ -140,6 +140,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			AssociatePublicIpAddress: b.config.AssociatePublicIpAddress,
 			RegionId:                 b.config.AlicloudRegion,
 			InternetChargeType:       b.config.InternetChargeType,
+			InternetMaxBandwidthOut:  b.config.InternetMaxBandwidthOut,
 		})
 	} else {
 		steps = append(steps, &stepConfigAlicloudPublicIP{

builder/alicloud/ecs/step_check_source_image.go

@@ -18,8 +18,12 @@ func (s *stepCheckAlicloudSourceImage) Run(_ context.Context, state multistep.St
 	client := state.Get("client").(*ecs.Client)
 	config := state.Get("config").(Config)
 	ui := state.Get("ui").(packer.Ui)
-	images, _, err := client.DescribeImages(&ecs.DescribeImagesArgs{RegionId: common.Region(config.AlicloudRegion),
+	args := &ecs.DescribeImagesArgs{
-		ImageId: config.AlicloudSourceImage})
+		RegionId: common.Region(config.AlicloudRegion),
+		ImageId:  config.AlicloudSourceImage,
+	}
+	args.PageSize = 50
+	images, _, err := client.DescribeImages(args)
 	if err != nil {
 		err := fmt.Errorf("Error querying alicloud image: %s", err)
 		state.Put("error", err)
@@ -27,6 +31,19 @@ func (s *stepCheckAlicloudSourceImage) Run(_ context.Context, state multistep.St
 		return multistep.ActionHalt
 	}
 
+	// Describe markerplace image
+	args.ImageOwnerAlias = ecs.ImageOwnerMarketplace
+	imageMarkets, _, err := client.DescribeImages(args)
+	if err != nil {
+		err := fmt.Errorf("Error querying alicloud marketplace image: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+	if len(imageMarkets) > 0 {
+		images = append(images, imageMarkets...)
+	}
+
 	if len(images) == 0 {
 		err := fmt.Errorf("No alicloud image was found matching filters: %v", config.AlicloudSourceImage)
 		state.Put("error", err)

builder/alicloud/ecs/step_config_eip.go

@@ -14,6 +14,7 @@ type stepConfigAlicloudEIP struct {
 	AssociatePublicIpAddress bool
 	RegionId                 string
 	InternetChargeType       string
+	InternetMaxBandwidthOut  int
 	allocatedId              string
 }
 
@@ -24,6 +25,7 @@ func (s *stepConfigAlicloudEIP) Run(_ context.Context, state multistep.StateBag)
 	ui.Say("Allocating eip")
 	ipaddress, allocateId, err := client.AllocateEipAddress(&ecs.AllocateEipAddressArgs{
 		RegionId: common.Region(s.RegionId), InternetChargeType: common.InternetChargeType(s.InternetChargeType),
+		Bandwidth: s.InternetMaxBandwidthOut,
 	})
 	if err != nil {
 		state.Put("error", err)

builder/alicloud/ecs/step_config_vpc.go

@@ -85,7 +85,8 @@ func (s *stepConfigAlicloudVPC) Cleanup(state multistep.StateBag) {
 			e, _ := err.(*common.Error)
 			if (e.Code == "DependencyViolation.Instance" || e.Code == "DependencyViolation.RouteEntry" ||
 				e.Code == "DependencyViolation.VSwitch" ||
-				e.Code == "DependencyViolation.SecurityGroup") && time.Now().Before(timeoutPoint) {
+				e.Code == "DependencyViolation.SecurityGroup" ||
+				e.Code == "Forbbiden") && time.Now().Before(timeoutPoint) {
 				time.Sleep(1 * time.Second)
 				continue
 			}

builder/amazon/chroot/builder.go

@@ -44,6 +44,7 @@ type Config struct {
 	RootVolumeSize    int64                      `mapstructure:"root_volume_size"`
 	SourceAmi         string                     `mapstructure:"source_ami"`
 	SourceAmiFilter   awscommon.AmiFilterOptions `mapstructure:"source_ami_filter"`
+	RootVolumeTags    awscommon.TagMap           `mapstructure:"root_volume_tags"`
 
 	ctx interpolate.Context
 }
@@ -67,6 +68,7 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 				"ami_description",
 				"snapshot_tags",
 				"tags",
+				"root_volume_tags",
 				"command_wrapper",
 				"post_mount_commands",
 				"pre_mount_commands",
@@ -230,6 +232,8 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 		&StepPrepareDevice{},
 		&StepCreateVolume{
 			RootVolumeSize: b.config.RootVolumeSize,
+			RootVolumeTags: b.config.RootVolumeTags,
+			Ctx:            b.config.ctx,
 		},
 		&StepAttachVolume{},
 		&StepEarlyUnflock{},

builder/amazon/chroot/step_attach_volume.go

@@ -2,11 +2,10 @@ package chroot
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"strings"
-	"time"
 
+	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	awscommon "github.com/hashicorp/packer/builder/amazon/common"
 	"github.com/hashicorp/packer/helper/multistep"
@@ -24,7 +23,7 @@ type StepAttachVolume struct {
 	volumeId string
 }
 
-func (s *StepAttachVolume) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepAttachVolume) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	device := state.Get("device").(string)
 	instance := state.Get("instance").(*ec2.Instance)
@@ -52,35 +51,7 @@ func (s *StepAttachVolume) Run(_ context.Context, state multistep.StateBag) mult
 	s.volumeId = volumeId
 
 	// Wait for the volume to become attached
-	stateChange := awscommon.StateChangeConf{
+	err = awscommon.WaitUntilVolumeAttached(ctx, ec2conn, s.volumeId)
-		Pending:   []string{"attaching"},
-		StepState: state,
-		Target:    "attached",
-		Refresh: func() (interface{}, string, error) {
-			attempts := 0
-			for attempts < 30 {
-				resp, err := ec2conn.DescribeVolumes(&ec2.DescribeVolumesInput{VolumeIds: []*string{&volumeId}})
-				if err != nil {
-					return nil, "", err
-				}
-				if len(resp.Volumes[0].Attachments) > 0 {
-					a := resp.Volumes[0].Attachments[0]
-					return a, *a.State, nil
-				}
-				// When Attachment on volume is not present sleep for 2s and retry
-				attempts += 1
-				ui.Say(fmt.Sprintf(
-					"Volume %s show no attachments. Attempt %d/30. Sleeping for 2s and will retry.",
-					volumeId, attempts))
-				time.Sleep(2 * time.Second)
-			}
-
-			// Attachment on volume is not present after all attempts
-			return nil, "", errors.New("No attachments on volume.")
-		},
-	}
-
-	_, err = awscommon.WaitForState(&stateChange)
 	if err != nil {
 		err := fmt.Errorf("Error waiting for volume: %s", err)
 		state.Put("error", err)
@@ -116,26 +87,7 @@ func (s *StepAttachVolume) CleanupFunc(state multistep.StateBag) error {
 	s.attached = false
 
 	// Wait for the volume to detach
-	stateChange := awscommon.StateChangeConf{
+	err = awscommon.WaitUntilVolumeDetached(aws.BackgroundContext(), ec2conn, s.volumeId)
-		Pending:   []string{"attaching", "attached", "detaching"},
-		StepState: state,
-		Target:    "detached",
-		Refresh: func() (interface{}, string, error) {
-			resp, err := ec2conn.DescribeVolumes(&ec2.DescribeVolumesInput{VolumeIds: []*string{&s.volumeId}})
-			if err != nil {
-				return nil, "", err
-			}
-
-			v := resp.Volumes[0]
-			if len(v.Attachments) > 0 {
-				return v, *v.Attachments[0].State, nil
-			} else {
-				return v, "detached", nil
-			}
-		},
-	}
-
-	_, err = awscommon.WaitForState(&stateChange)
 	if err != nil {
 		return fmt.Errorf("Error waiting for volume: %s", err)
 	}

builder/amazon/chroot/step_create_volume.go

@@ -10,6 +10,7 @@ import (
 	awscommon "github.com/hashicorp/packer/builder/amazon/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
+	"github.com/hashicorp/packer/template/interpolate"
 )
 
 // StepCreateVolume creates a new volume from the snapshot of the root
@@ -20,14 +21,36 @@ import (
 type StepCreateVolume struct {
 	volumeId       string
 	RootVolumeSize int64
+	RootVolumeTags awscommon.TagMap
+	Ctx            interpolate.Context
 }
 
-func (s *StepCreateVolume) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepCreateVolume) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(*Config)
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	instance := state.Get("instance").(*ec2.Instance)
 	ui := state.Get("ui").(packer.Ui)
 
+	volTags, err := s.RootVolumeTags.EC2Tags(s.Ctx, *ec2conn.Config.Region, state)
+	if err != nil {
+		err := fmt.Errorf("Error tagging volumes: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+
+	// Collect tags for tagging on resource creation
+	var tagSpecs []*ec2.TagSpecification
+
+	if len(volTags) > 0 {
+		runVolTags := &ec2.TagSpecification{
+			ResourceType: aws.String("volume"),
+			Tags:         volTags,
+		}
+
+		tagSpecs = append(tagSpecs, runVolTags)
+	}
+
 	var createVolume *ec2.CreateVolumeInput
 	if config.FromScratch {
 		createVolume = &ec2.CreateVolumeInput{
@@ -69,6 +92,10 @@ func (s *StepCreateVolume) Run(_ context.Context, state multistep.StateBag) mult
 		}
 	}
 
+	if len(tagSpecs) > 0 {
+		createVolume.SetTagSpecifications(tagSpecs)
+		volTags.Report(ui)
+	}
 	log.Printf("Create args: %+v", createVolume)
 
 	createVolumeResp, err := ec2conn.CreateVolume(createVolume)
@@ -84,22 +111,7 @@ func (s *StepCreateVolume) Run(_ context.Context, state multistep.StateBag) mult
 	log.Printf("Volume ID: %s", s.volumeId)
 
 	// Wait for the volume to become ready
-	stateChange := awscommon.StateChangeConf{
+	err = awscommon.WaitUntilVolumeAvailable(ctx, ec2conn, s.volumeId)
-		Pending:   []string{"creating"},
-		StepState: state,
-		Target:    "available",
-		Refresh: func() (interface{}, string, error) {
-			resp, err := ec2conn.DescribeVolumes(&ec2.DescribeVolumesInput{VolumeIds: []*string{&s.volumeId}})
-			if err != nil {
-				return nil, "", err
-			}
-
-			v := resp.Volumes[0]
-			return v, *v.State, nil
-		},
-	}
-
-	_, err = awscommon.WaitForState(&stateChange)
 	if err != nil {
 		err := fmt.Errorf("Error waiting for volume: %s", err)
 		state.Put("error", err)

builder/amazon/chroot/step_register_ami.go

@@ -18,7 +18,7 @@ type StepRegisterAMI struct {
 	EnableAMISriovNetSupport bool
 }
 
-func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepRegisterAMI) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(*Config)
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	snapshotId := state.Get("snapshot_id").(string)
@@ -102,16 +102,8 @@ func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multi
 	amis[*ec2conn.Config.Region] = *registerResp.ImageId
 	state.Put("amis", amis)
 
-	// Wait for the image to become ready
-	stateChange := awscommon.StateChangeConf{
-		Pending:   []string{"pending"},
-		Target:    "available",
-		Refresh:   awscommon.AMIStateRefreshFunc(ec2conn, *registerResp.ImageId),
-		StepState: state,
-	}
-
 	ui.Say("Waiting for AMI to become ready...")
-	if _, err := awscommon.WaitForState(&stateChange); err != nil {
+	if err := awscommon.WaitUntilAMIAvailable(ctx, ec2conn, *registerResp.ImageId); err != nil {
 		err := fmt.Errorf("Error waiting for AMI: %s", err)
 		state.Put("error", err)
 		ui.Error(err.Error())

builder/amazon/chroot/step_snapshot.go

@@ -2,7 +2,6 @@ package chroot
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"time"
 
@@ -20,7 +19,7 @@ type StepSnapshot struct {
 	snapshotId string
 }
 
-func (s *StepSnapshot) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepSnapshot) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	ui := state.Get("ui").(packer.Ui)
 	volumeId := state.Get("volume_id").(string)
@@ -44,26 +43,7 @@ func (s *StepSnapshot) Run(_ context.Context, state multistep.StateBag) multiste
 	ui.Message(fmt.Sprintf("Snapshot ID: %s", s.snapshotId))
 
 	// Wait for the snapshot to be ready
-	stateChange := awscommon.StateChangeConf{
+	err = awscommon.WaitUntilSnapshotDone(ctx, ec2conn, s.snapshotId)
-		Pending:   []string{"pending"},
-		StepState: state,
-		Target:    "completed",
-		Refresh: func() (interface{}, string, error) {
-			resp, err := ec2conn.DescribeSnapshots(&ec2.DescribeSnapshotsInput{SnapshotIds: []*string{&s.snapshotId}})
-			if err != nil {
-				return nil, "", err
-			}
-
-			if len(resp.Snapshots) == 0 {
-				return nil, "", errors.New("No snapshots found.")
-			}
-
-			s := resp.Snapshots[0]
-			return s, *s.State, nil
-		},
-	}
-
-	_, err = awscommon.WaitForState(&stateChange)
 	if err != nil {
 		err := fmt.Errorf("Error waiting for snapshot: %s", err)
 		state.Put("error", err)

builder/amazon/common/run_config.go

@@ -25,6 +25,10 @@ func (d *AmiFilterOptions) Empty() bool {
 	return len(d.Owners) == 0 && len(d.Filters) == 0
 }
 
+func (d *AmiFilterOptions) NoOwner() bool {
+	return len(d.Owners) == 0
+}
+
 // RunConfig contains configuration for running an instance from a source
 // AMI and details on how to access that launched image.
 type RunConfig struct {
@@ -43,6 +47,7 @@ type RunConfig struct {
 	SourceAmiFilter                   AmiFilterOptions  `mapstructure:"source_ami_filter"`
 	SpotPrice                         string            `mapstructure:"spot_price"`
 	SpotPriceAutoProduct              string            `mapstructure:"spot_price_auto_product"`
+	SpotTags                          map[string]string `mapstructure:"spot_tags"`
 	SubnetId                          string            `mapstructure:"subnet_id"`
 	TemporaryKeyPairName              string            `mapstructure:"temporary_key_pair_name"`
 	TemporarySGSourceCidr             string            `mapstructure:"temporary_security_group_source_cidr"`
@@ -100,6 +105,10 @@ func (c *RunConfig) Prepare(ctx *interpolate.Context) []error {
 		errs = append(errs, fmt.Errorf("A source_ami or source_ami_filter must be specified"))
 	}
 
+	if c.SourceAmi == "" && c.SourceAmiFilter.NoOwner() {
+		errs = append(errs, fmt.Errorf("For security reasons, your source AMI filter must declare an owner."))
+	}
+
 	if c.InstanceType == "" {
 		errs = append(errs, fmt.Errorf("An instance_type must be specified"))
 	}
@@ -118,6 +127,13 @@ func (c *RunConfig) Prepare(ctx *interpolate.Context) []error {
 		}
 	}
 
+	if c.SpotTags != nil {
+		if c.SpotPrice == "" || c.SpotPrice == "0" {
+			errs = append(errs, fmt.Errorf(
+				"spot_tags should not be set when not requesting a spot instance"))
+		}
+	}
+
 	if c.UserData != "" && c.UserDataFile != "" {
 		errs = append(errs, fmt.Errorf("Only one of user_data or user_data_file can be specified."))
 	} else if c.UserDataFile != "" {

builder/amazon/common/run_config_test.go

@@ -55,18 +55,28 @@ func TestRunConfigPrepare_InstanceType(t *testing.T) {
 func TestRunConfigPrepare_SourceAmi(t *testing.T) {
 	c := testConfig()
 	c.SourceAmi = ""
-	if err := c.Prepare(nil); len(err) != 1 {
+	if err := c.Prepare(nil); len(err) != 2 {
 		t.Fatalf("Should error if a source_ami (or source_ami_filter) is not specified")
 	}
 }
 
 func TestRunConfigPrepare_SourceAmiFilterBlank(t *testing.T) {
 	c := testConfigFilter()
-	if err := c.Prepare(nil); len(err) != 1 {
+	if err := c.Prepare(nil); len(err) != 2 {
 		t.Fatalf("Should error if source_ami_filter is empty or not specified (and source_ami is not specified)")
 	}
 }
 
+func TestRunConfigPrepare_SourceAmiFilterOwnersBlank(t *testing.T) {
+	c := testConfigFilter()
+	filter_key := "name"
+	filter_value := "foo"
+	c.SourceAmiFilter = AmiFilterOptions{Filters: map[*string]*string{&filter_key: &filter_value}}
+	if err := c.Prepare(nil); len(err) != 1 {
+		t.Fatalf("Should error if Owners is not specified)")
+	}
+}
+
 func TestRunConfigPrepare_SourceAmiFilterGood(t *testing.T) {
 	c := testConfigFilter()
 	owner := "123"

builder/amazon/common/state.go

@@ -1,16 +1,13 @@
 package common
 
 import (
-	"errors"
-	"fmt"
 	"log"
-	"net"
 	"os"
 	"strconv"
-	"strings"
 	"time"
 
-	"github.com/aws/aws-sdk-go/aws/awserr"
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/request"
 	"github.com/aws/aws-sdk-go/service/ec2"
 	"github.com/hashicorp/packer/helper/multistep"
 )
@@ -35,228 +32,304 @@ type StateChangeConf struct {
 	Target    string
 }
 
-// AMIStateRefreshFunc returns a StateRefreshFunc that is used to watch
+// Following are wrapper functions that use Packer's environment-variables to
-// an AMI for state changes.
+// determing retry logic, then call the AWS SDK's built-in waiters.
-func AMIStateRefreshFunc(conn *ec2.EC2, imageId string) StateRefreshFunc {
-	return func() (interface{}, string, error) {
-		resp, err := conn.DescribeImages(&ec2.DescribeImagesInput{
-			ImageIds: []*string{&imageId},
-		})
-		if err != nil {
-			if ec2err, ok := err.(awserr.Error); ok && ec2err.Code() == "InvalidAMIID.NotFound" {
-				// Set this to nil as if we didn't find anything.
-				resp = nil
-			} else if isTransientNetworkError(err) {
-				// Transient network error, treat it as if we didn't find anything
-				resp = nil
-			} else {
-				log.Printf("Error on AMIStateRefresh: %s", err)
-				return nil, "", err
-			}
-		}
 
-		if resp == nil || len(resp.Images) == 0 {
+func WaitUntilAMIAvailable(ctx aws.Context, conn *ec2.EC2, imageId string) error {
-			// Sometimes AWS has consistency issues and doesn't see the
+	imageInput := ec2.DescribeImagesInput{
-			// AMI. Return an empty state.
+		ImageIds: []*string{&imageId},
-			return nil, "", nil
-		}
-
-		i := resp.Images[0]
-		return i, *i.State, nil
 	}
+
+	err := conn.WaitUntilImageAvailableWithContext(
+		ctx,
+		&imageInput,
+		getWaiterOptions()...)
+	return err
 }
 
-// InstanceStateRefreshFunc returns a StateRefreshFunc that is used to watch
+func WaitUntilInstanceTerminated(ctx aws.Context, conn *ec2.EC2, instanceId string) error {
-// an EC2 instance.
-func InstanceStateRefreshFunc(conn *ec2.EC2, instanceId string) StateRefreshFunc {
-	return func() (interface{}, string, error) {
-		resp, err := conn.DescribeInstances(&ec2.DescribeInstancesInput{
-			InstanceIds: []*string{&instanceId},
-		})
-		if err != nil {
-			if ec2err, ok := err.(awserr.Error); ok && ec2err.Code() == "InvalidInstanceID.NotFound" {
-				// Set this to nil as if we didn't find anything.
-				resp = nil
-			} else if isTransientNetworkError(err) {
-				// Transient network error, treat it as if we didn't find anything
-				resp = nil
-			} else {
-				log.Printf("Error on InstanceStateRefresh: %s", err)
-				return nil, "", err
-			}
-		}
 
-		if resp == nil || len(resp.Reservations) == 0 || len(resp.Reservations[0].Instances) == 0 {
+	instanceInput := ec2.DescribeInstancesInput{
-			// Sometimes AWS just has consistency issues and doesn't see
+		InstanceIds: []*string{&instanceId},
-			// our instance yet. Return an empty state.
-			return nil, "", nil
-		}
-
-		i := resp.Reservations[0].Instances[0]
-		return i, *i.State.Name, nil
 	}
+
+	err := conn.WaitUntilInstanceTerminatedWithContext(
+		ctx,
+		&instanceInput,
+		getWaiterOptions()...)
+	return err
 }
 
-// SpotRequestStateRefreshFunc returns a StateRefreshFunc that is used to watch
+// This function works for both requesting and cancelling spot instances.
-// a spot request for state changes.
+func WaitUntilSpotRequestFulfilled(ctx aws.Context, conn *ec2.EC2, spotRequestId string) error {
-func SpotRequestStateRefreshFunc(conn *ec2.EC2, spotRequestId string) StateRefreshFunc {
+	spotRequestInput := ec2.DescribeSpotInstanceRequestsInput{
-	return func() (interface{}, string, error) {
+		SpotInstanceRequestIds: []*string{&spotRequestId},
-		resp, err := conn.DescribeSpotInstanceRequests(&ec2.DescribeSpotInstanceRequestsInput{
-			SpotInstanceRequestIds: []*string{&spotRequestId},
-		})
-
-		if err != nil {
-			if ec2err, ok := err.(awserr.Error); ok && ec2err.Code() == "InvalidSpotInstanceRequestID.NotFound" {
-				// Set this to nil as if we didn't find anything.
-				resp = nil
-			} else if isTransientNetworkError(err) {
-				// Transient network error, treat it as if we didn't find anything
-				resp = nil
-			} else {
-				log.Printf("Error on SpotRequestStateRefresh: %s", err)
-				return nil, "", err
-			}
-		}
-
-		if resp == nil || len(resp.SpotInstanceRequests) == 0 {
-			// Sometimes AWS has consistency issues and doesn't see the
-			// SpotRequest. Return an empty state.
-			return nil, "", nil
-		}
-
-		i := resp.SpotInstanceRequests[0]
-		return i, *i.State, nil
 	}
+
+	err := conn.WaitUntilSpotInstanceRequestFulfilledWithContext(
+		ctx,
+		&spotRequestInput,
+		getWaiterOptions()...)
+	return err
 }
 
-func ImportImageRefreshFunc(conn *ec2.EC2, importTaskId string) StateRefreshFunc {
+func WaitUntilVolumeAvailable(ctx aws.Context, conn *ec2.EC2, volumeId string) error {
-	return func() (interface{}, string, error) {
+	volumeInput := ec2.DescribeVolumesInput{
-		resp, err := conn.DescribeImportImageTasks(&ec2.DescribeImportImageTasksInput{
+		VolumeIds: []*string{&volumeId},
-			ImportTaskIds: []*string{
+	}
-				&importTaskId,
+
+	err := conn.WaitUntilVolumeAvailableWithContext(
+		ctx,
+		&volumeInput,
+		getWaiterOptions()...)
+	return err
+}
+
+func WaitUntilSnapshotDone(ctx aws.Context, conn *ec2.EC2, snapshotID string) error {
+	snapInput := ec2.DescribeSnapshotsInput{
+		SnapshotIds: []*string{&snapshotID},
+	}
+
+	err := conn.WaitUntilSnapshotCompletedWithContext(
+		ctx,
+		&snapInput,
+		getWaiterOptions()...)
+	return err
+}
+
+// Wrappers for our custom AWS waiters
+
+func WaitUntilVolumeAttached(ctx aws.Context, conn *ec2.EC2, volumeId string) error {
+	volumeInput := ec2.DescribeVolumesInput{
+		VolumeIds: []*string{&volumeId},
+	}
+
+	err := WaitForVolumeToBeAttached(conn,
+		ctx,
+		&volumeInput,
+		getWaiterOptions()...)
+	return err
+}
+
+func WaitUntilVolumeDetached(ctx aws.Context, conn *ec2.EC2, volumeId string) error {
+	volumeInput := ec2.DescribeVolumesInput{
+		VolumeIds: []*string{&volumeId},
+	}
+
+	err := WaitForVolumeToBeDetached(conn,
+		ctx,
+		&volumeInput,
+		getWaiterOptions()...)
+	return err
+}
+
+func WaitUntilImageImported(ctx aws.Context, conn *ec2.EC2, taskID string) error {
+	importInput := ec2.DescribeImportImageTasksInput{
+		ImportTaskIds: []*string{&taskID},
+	}
+
+	err := WaitForImageToBeImported(conn,
+		ctx,
+		&importInput,
+		getWaiterOptions()...)
+	return err
+}
+
+// Custom waiters using AWS's request.Waiter
+
+func WaitForVolumeToBeAttached(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeVolumesInput, opts ...request.WaiterOption) error {
+	w := request.Waiter{
+		Name:        "DescribeVolumes",
+		MaxAttempts: 40,
+		Delay:       request.ConstantWaiterDelay(5 * time.Second),
+		Acceptors: []request.WaiterAcceptor{
+			{
+				State:    request.SuccessWaiterState,
+				Matcher:  request.PathAllWaiterMatch,
+				Argument: "Volumes[].Attachments[].State",
+				Expected: "attached",
 			},
 		},
-		)
+		Logger: c.Config.Logger,
-		if err != nil {
+		NewRequest: func(opts []request.Option) (*request.Request, error) {
-			if ec2err, ok := err.(awserr.Error); ok && strings.HasPrefix(ec2err.Code(), "InvalidConversionTaskId") {
+			var inCpy *ec2.DescribeVolumesInput
-				resp = nil
+			if input != nil {
-			} else if isTransientNetworkError(err) {
+				tmp := *input
-				resp = nil
+				inCpy = &tmp
-			} else {
-				log.Printf("Error on ImportImageRefresh: %s", err)
-				return nil, "", err
 			}
-		}
+			req, _ := c.DescribeVolumesRequest(inCpy)
-
+			req.SetContext(ctx)
-		if resp == nil || len(resp.ImportImageTasks) == 0 {
+			req.ApplyOptions(opts...)
-			return nil, "", nil
+			return req, nil
-		}
+		},
-
-		i := resp.ImportImageTasks[0]
-		return i, *i.Status, nil
 	}
+	return w.WaitWithContext(ctx)
 }
 
-// WaitForState watches an object and waits for it to achieve a certain
+func WaitForVolumeToBeDetached(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeVolumesInput, opts ...request.WaiterOption) error {
-// state.
+	w := request.Waiter{
-func WaitForState(conf *StateChangeConf) (i interface{}, err error) {
+		Name:        "DescribeVolumes",
-	log.Printf("Waiting for state to become: %s", conf.Target)
+		MaxAttempts: 40,
-
+		Delay:       request.ConstantWaiterDelay(5 * time.Second),
-	sleepSeconds := SleepSeconds()
+		Acceptors: []request.WaiterAcceptor{
-	maxTicks := TimeoutSeconds()/sleepSeconds + 1
+			{
-	notfoundTick := 0
+				State:    request.SuccessWaiterState,
-
+				Matcher:  request.PathAllWaiterMatch,
-	for {
+				Argument: "length(Volumes[].Attachments[]) == `0`",
-		var currentState string
+				Expected: true,
-		i, currentState, err = conf.Refresh()
+			},
-		if err != nil {
+		},
-			return
+		Logger: c.Config.Logger,
-		}
+		NewRequest: func(opts []request.Option) (*request.Request, error) {
-
+			var inCpy *ec2.DescribeVolumesInput
-		if i == nil {
+			if input != nil {
-			// If we didn't find the resource, check if we have been
+				tmp := *input
-			// not finding it for awhile, and if so, report an error.
+				inCpy = &tmp
-			notfoundTick += 1
-			if notfoundTick > maxTicks {
-				return nil, errors.New("couldn't find resource")
 			}
-		} else {
+			req, _ := c.DescribeVolumesRequest(inCpy)
-			// Reset the counter for when a resource isn't found
+			req.SetContext(ctx)
-			notfoundTick = 0
+			req.ApplyOptions(opts...)
-
+			return req, nil
-			if currentState == conf.Target {
+		},
-				return
-			}
-
-			if conf.StepState != nil {
-				if _, ok := conf.StepState.GetOk(multistep.StateCancelled); ok {
-					return nil, errors.New("interrupted")
-				}
-			}
-
-			found := false
-			for _, allowed := range conf.Pending {
-				if currentState == allowed {
-					found = true
-					break
-				}
-			}
-
-			if !found {
-				err := fmt.Errorf("unexpected state '%s', wanted target '%s'", currentState, conf.Target)
-				return nil, err
-			}
-		}
-
-		time.Sleep(time.Duration(sleepSeconds) * time.Second)
 	}
+	return w.WaitWithContext(ctx)
 }
 
-func isTransientNetworkError(err error) bool {
+func WaitForImageToBeImported(c *ec2.EC2, ctx aws.Context, input *ec2.DescribeImportImageTasksInput, opts ...request.WaiterOption) error {
-	if nerr, ok := err.(net.Error); ok && nerr.Temporary() {
+	w := request.Waiter{
-		return true
+		Name:        "DescribeImages",
+		MaxAttempts: 300,
+		Delay:       request.ConstantWaiterDelay(5 * time.Second),
+		Acceptors: []request.WaiterAcceptor{
+			{
+				State:    request.SuccessWaiterState,
+				Matcher:  request.PathAllWaiterMatch,
+				Argument: "ImportImageTasks[].Status",
+				Expected: "completed",
+			},
+		},
+		Logger: c.Config.Logger,
+		NewRequest: func(opts []request.Option) (*request.Request, error) {
+			var inCpy *ec2.DescribeImportImageTasksInput
+			if input != nil {
+				tmp := *input
+				inCpy = &tmp
+			}
+			req, _ := c.DescribeImportImageTasksRequest(inCpy)
+			req.SetContext(ctx)
+			req.ApplyOptions(opts...)
+			return req, nil
+		},
 	}
-
+	return w.WaitWithContext(ctx)
-	return false
 }
 
-// Returns 300 seconds (5 minutes) by default
+// This helper function uses the environment variables AWS_TIMEOUT_SECONDS and
-// Some AWS operations, like copying an AMI to a distant region, take a very long time
+// AWS_POLL_DELAY_SECONDS to generate waiter options that can be passed into any
-// Allow user to override with AWS_TIMEOUT_SECONDS environment variable
+// request.Waiter function. These options will control how many times the waiter
-func TimeoutSeconds() (seconds int) {
+// will retry the request, as well as how long to wait between the retries.
-	seconds = 300
 
-	override := os.Getenv("AWS_TIMEOUT_SECONDS")
+// DEFAULTING BEHAVIOR:
+// if AWS_POLL_DELAY_SECONDS is set but the others are not, Packer will set this
+// poll delay and use the waiter-specific default
+
+// if AWS_TIMEOUT_SECONDS is set but AWS_MAX_ATTEMPTS is not, Packer will use
+// AWS_TIMEOUT_SECONDS and _either_ AWS_POLL_DELAY_SECONDS _or_ 2 if the user has not set AWS_POLL_DELAY_SECONDS, to determine a max number of attempts to make.
+
+// if AWS_TIMEOUT_SECONDS, _and_ AWS_MAX_ATTEMPTS are both set,
+// AWS_TIMEOUT_SECONDS will be ignored.
+
+// if AWS_MAX_ATTEMPTS is set but AWS_POLL_DELAY_SECONDS is not, then we will
+// use waiter-specific defaults.
+
+type envInfo struct {
+	envKey     string
+	Val        int
+	overridden bool
+}
+
+type overridableWaitVars struct {
+	awsPollDelaySeconds envInfo
+	awsMaxAttempts      envInfo
+	awsTimeoutSeconds   envInfo
+}
+
+func getWaiterOptions() []request.WaiterOption {
+	envOverrides := getEnvOverrides()
+	waitOpts := applyEnvOverrides(envOverrides)
+	return waitOpts
+}
+
+func getOverride(varInfo envInfo) envInfo {
+	override := os.Getenv(varInfo.envKey)
 	if override != "" {
 		n, err := strconv.Atoi(override)
 		if err != nil {
-			log.Printf("Invalid timeout seconds '%s', using default", override)
+			log.Printf("Invalid %s '%s', using default", varInfo.envKey, override)
 		} else {
-			seconds = n
+			varInfo.overridden = true
+			varInfo.Val = n
 		}
 	}
 
-	log.Printf("Allowing %ds to complete (change with AWS_TIMEOUT_SECONDS)", seconds)
+	return varInfo
-	return seconds
 }
-
+func getEnvOverrides() overridableWaitVars {
-// Returns 2 seconds by default
+	// Load env vars from environment, and use them to override defaults
-// AWS async operations sometimes takes long times, if there are multiple parallel builds,
+	envValues := overridableWaitVars{
-// polling at 2 second frequency will exceed the request limit. Allow 2 seconds to be
+		envInfo{"AWS_POLL_DELAY_SECONDS", 2, false},
-// overwritten with AWS_POLL_DELAY_SECONDS
+		envInfo{"AWS_MAX_ATTEMPTS", 0, false},
-func SleepSeconds() (seconds int) {
+		envInfo{"AWS_TIMEOUT_SECONDS", 300, false},
-	seconds = 2
-
-	override := os.Getenv("AWS_POLL_DELAY_SECONDS")
-	if override != "" {
-		n, err := strconv.Atoi(override)
-		if err != nil {
-			log.Printf("Invalid sleep seconds '%s', using default", override)
-		} else {
-			seconds = n
-		}
 	}
 
-	log.Printf("Using %ds as polling delay (change with AWS_POLL_DELAY_SECONDS)", seconds)
+	envValues.awsMaxAttempts = getOverride(envValues.awsMaxAttempts)
-	return seconds
+	envValues.awsPollDelaySeconds = getOverride(envValues.awsPollDelaySeconds)
+	envValues.awsTimeoutSeconds = getOverride(envValues.awsTimeoutSeconds)
+
+	return envValues
+}
+
+func applyEnvOverrides(envOverrides overridableWaitVars) []request.WaiterOption {
+	waitOpts := make([]request.WaiterOption, 0)
+	// If user has set poll delay seconds, overwrite it. If user has NOT,
+	// default to a poll delay of 2 seconds
+	if envOverrides.awsPollDelaySeconds.overridden {
+		delaySeconds := request.ConstantWaiterDelay(time.Duration(envOverrides.awsPollDelaySeconds.Val) * time.Second)
+		waitOpts = append(waitOpts, request.WithWaiterDelay(delaySeconds))
+	}
+
+	// If user has set max attempts, overwrite it. If user hasn't set max
+	// attempts, default to whatever the waiter has set as a default.
+	if envOverrides.awsMaxAttempts.overridden {
+		waitOpts = append(waitOpts, request.WithWaiterMaxAttempts(envOverrides.awsMaxAttempts.Val))
+	}
+
+	if envOverrides.awsMaxAttempts.overridden && envOverrides.awsTimeoutSeconds.overridden {
+		log.Printf("WARNING: AWS_MAX_ATTEMPTS and AWS_TIMEOUT_SECONDS are" +
+			" both set. Packer will be using AWS_MAX_ATTEMPTS and discarding " +
+			"AWS_TIMEOUT_SECONDS. If you have not set AWS_POLL_DELAY_SECONDS, " +
+			"Packer will default to a 2 second poll delay.")
+	} else if envOverrides.awsTimeoutSeconds.overridden {
+		log.Printf("DEPRECATION WARNING: env var AWS_TIMEOUT_SECONDS is " +
+			"deprecated in favor of AWS_MAX_ATTEMPTS. If you have not " +
+			"explicitly set AWS_POLL_DELAY_SECONDS, we are defaulting to a " +
+			"poll delay of 2 seconds, regardless of the AWS waiter's default.")
+		maxAttempts := envOverrides.awsTimeoutSeconds.Val / envOverrides.awsPollDelaySeconds.Val
+		// override the delay so we can get the timeout right
+		if !envOverrides.awsPollDelaySeconds.overridden {
+			delaySeconds := request.ConstantWaiterDelay(time.Duration(envOverrides.awsPollDelaySeconds.Val) * time.Second)
+			waitOpts = append(waitOpts, request.WithWaiterDelay(delaySeconds))
+		}
+		waitOpts = append(waitOpts, request.WithWaiterMaxAttempts(maxAttempts))
+	}
+	if len(waitOpts) == 0 {
+		log.Printf("No AWS timeout and polling overrides have been set. " +
+			"Packer will default to waiter-specific delays and timeouts. If you would " +
+			"like to customize the length of time between retries and max " +
+			"number of retries you may do so by setting the environment " +
+			"variables AWS_POLL_DELAY_SECONDS and AWS_MAX_ATTEMPTS to your " +
+			"desired values.")
+	}
+
+	return waitOpts
 }

builder/amazon/common/state_test.go

@@ -0,0 +1,66 @@
+package common
+
+import (
+	"reflect"
+	"testing"
+	"time"
+
+	"github.com/aws/aws-sdk-go/aws/request"
+)
+
+func testGetWaiterOptions(t *testing.T) {
+	// no vars are set
+	envValues := overridableWaitVars{
+		envInfo{"AWS_POLL_DELAY_SECONDS", 2, false},
+		envInfo{"AWS_MAX_ATTEMPTS", 0, false},
+		envInfo{"AWS_TIMEOUT_SECONDS", 300, false},
+	}
+	options := applyEnvOverrides(envValues)
+	if len(options) > 0 {
+		t.Fatalf("Did not expect any waiter options to be generated; actual: %#v", options)
+	}
+
+	// all vars are set
+	envValues = overridableWaitVars{
+		envInfo{"AWS_POLL_DELAY_SECONDS", 1, true},
+		envInfo{"AWS_MAX_ATTEMPTS", 800, true},
+		envInfo{"AWS_TIMEOUT_SECONDS", 20, true},
+	}
+	options = applyEnvOverrides(envValues)
+	expected := []request.WaiterOption{
+		request.WithWaiterDelay(request.ConstantWaiterDelay(time.Duration(1) * time.Second)),
+		request.WithWaiterMaxAttempts(800),
+	}
+	if !reflect.DeepEqual(options, expected) {
+		t.Fatalf("expected != actual!! Expected: %#v; Actual: %#v.", expected, options)
+	}
+
+	// poll delay is not set
+	envValues = overridableWaitVars{
+		envInfo{"AWS_POLL_DELAY_SECONDS", 2, false},
+		envInfo{"AWS_MAX_ATTEMPTS", 800, true},
+		envInfo{"AWS_TIMEOUT_SECONDS", 300, false},
+	}
+	options = applyEnvOverrides(envValues)
+	expected = []request.WaiterOption{
+		request.WithWaiterMaxAttempts(800),
+	}
+	if !reflect.DeepEqual(options, expected) {
+		t.Fatalf("expected != actual!! Expected: %#v; Actual: %#v.", expected, options)
+	}
+
+	// poll delay is not set but timeout seconds is
+	envValues = overridableWaitVars{
+		envInfo{"AWS_POLL_DELAY_SECONDS", 2, false},
+		envInfo{"AWS_MAX_ATTEMPTS", 0, false},
+		envInfo{"AWS_TIMEOUT_SECONDS", 20, true},
+	}
+	options = applyEnvOverrides(envValues)
+	expected = []request.WaiterOption{
+		request.WithWaiterDelay(request.ConstantWaiterDelay(time.Duration(2) * time.Second)),
+		request.WithWaiterMaxAttempts(10),
+	}
+	if !reflect.DeepEqual(options, expected) {
+		t.Fatalf("expected != actual!! Expected: %#v; Actual: %#v.", expected, options)
+	}
+}

builder/amazon/common/step_ami_region_copy.go

@@ -20,7 +20,7 @@ type StepAMIRegionCopy struct {
 	Name              string
 }
 
-func (s *StepAMIRegionCopy) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepAMIRegionCopy) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	ui := state.Get("ui").(packer.Ui)
 	amis := state.Get("amis").(map[string]string)
@@ -53,7 +53,7 @@ func (s *StepAMIRegionCopy) Run(_ context.Context, state multistep.StateBag) mul
 
 		go func(region string) {
 			defer wg.Done()
-			id, snapshotIds, err := amiRegionCopy(state, s.AccessConfig, s.Name, ami, region, *ec2conn.Config.Region, regKeyID)
+			id, snapshotIds, err := amiRegionCopy(ctx, state, s.AccessConfig, s.Name, ami, region, *ec2conn.Config.Region, regKeyID)
 			lock.Lock()
 			defer lock.Unlock()
 			amis[region] = id
@@ -85,7 +85,7 @@ func (s *StepAMIRegionCopy) Cleanup(state multistep.StateBag) {
 
 // amiRegionCopy does a copy for the given AMI to the target region and
 // returns the resulting ID and snapshot IDs, or error.
-func amiRegionCopy(state multistep.StateBag, config *AccessConfig, name string, imageId string,
+func amiRegionCopy(ctx context.Context, state multistep.StateBag, config *AccessConfig, name string, imageId string,
 	target string, source string, keyID string) (string, []string, error) {
 	snapshotIds := []string{}
 	isEncrypted := false
@@ -116,14 +116,8 @@ func amiRegionCopy(state multistep.StateBag, config *AccessConfig, name string,
 			imageId, target, err)
 	}
 
-	stateChange := StateChangeConf{
+	// Wait for the image to become ready
-		Pending:   []string{"pending"},
+	if err := WaitUntilAMIAvailable(ctx, regionconn, *resp.ImageId); err != nil {
-		Target:    "available",
-		Refresh:   AMIStateRefreshFunc(regionconn, *resp.ImageId),
-		StepState: state,
-	}
-
-	if _, err := WaitForState(&stateChange); err != nil {
 		return "", snapshotIds, fmt.Errorf("Error waiting for AMI (%s) in region (%s): %s",
 			*resp.ImageId, target, err)
 	}

builder/amazon/common/step_cleanup_volumes.go

@@ -1,4 +1,4 @@
-package ebs
+package common
 
 import (
 	"context"
@@ -6,7 +6,6 @@ import (
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/ec2"
-	"github.com/hashicorp/packer/builder/amazon/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
@@ -14,16 +13,16 @@ import (
 // stepCleanupVolumes cleans up any orphaned volumes that were not designated to
 // remain after termination of the instance. These volumes are typically ones
 // that are marked as "delete on terminate:false" in the source_ami of a build.
-type stepCleanupVolumes struct {
+type StepCleanupVolumes struct {
-	BlockDevices common.BlockDevices
+	BlockDevices BlockDevices
 }
 
-func (s *stepCleanupVolumes) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepCleanupVolumes) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
 	// stepCleanupVolumes is for Cleanup only
 	return multistep.ActionContinue
 }
 
-func (s *stepCleanupVolumes) Cleanup(state multistep.StateBag) {
+func (s *StepCleanupVolumes) Cleanup(state multistep.StateBag) {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	instanceRaw := state.Get("instance")
 	var instance *ec2.Instance

builder/amazon/common/step_encrypted_ami.go

@@ -19,7 +19,7 @@ type StepCreateEncryptedAMICopy struct {
 	AMIMappings       []BlockDevice
 }
 
-func (s *StepCreateEncryptedAMICopy) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepCreateEncryptedAMICopy) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	ui := state.Get("ui").(packer.Ui)
 	kmsKeyId := s.KeyID
@@ -65,15 +65,8 @@ func (s *StepCreateEncryptedAMICopy) Run(_ context.Context, state multistep.Stat
 	}
 
 	// Wait for the copy to become ready
-	stateChange := StateChangeConf{
-		Pending:   []string{"pending"},
-		Target:    "available",
-		Refresh:   AMIStateRefreshFunc(ec2conn, *copyResp.ImageId),
-		StepState: state,
-	}
-
 	ui.Say("Waiting for AMI copy to become ready...")
-	if _, err := WaitForState(&stateChange); err != nil {
+	if err := WaitUntilAMIAvailable(ctx, ec2conn, *copyResp.ImageId); err != nil {
 		err := fmt.Errorf("Error waiting for AMI Copy: %s", err)
 		state.Put("error", err)
 		ui.Error(err.Error())

builder/amazon/common/step_run_source_instance.go

@@ -303,14 +303,8 @@ func (s *StepRunSourceInstance) Cleanup(state multistep.StateBag) {
 			ui.Error(fmt.Sprintf("Error terminating instance, may still be around: %s", err))
 			return
 		}
-		stateChange := StateChangeConf{
-			Pending: []string{"pending", "running", "shutting-down", "stopped", "stopping"},
-			Refresh: InstanceStateRefreshFunc(ec2conn, s.instanceId),
-			Target:  "terminated",
-		}
 
-		_, err := WaitForState(&stateChange)
+		if err := WaitUntilInstanceTerminated(aws.BackgroundContext(), ec2conn, s.instanceId); err != nil {
-		if err != nil {
 			ui.Error(err.Error())
 		}
 	}

builder/amazon/common/step_run_spot_instance.go

@@ -32,6 +32,7 @@ type StepRunSpotInstance struct {
 	SourceAMI                         string
 	SpotPrice                         string
 	SpotPriceProduct                  string
+	SpotTags                          TagMap
 	SubnetId                          string
 	Tags                              TagMap
 	VolumeTags                        TagMap
@@ -202,13 +203,7 @@ func (s *StepRunSpotInstance) Run(ctx context.Context, state multistep.StateBag)
 
 	spotRequestId := s.spotRequest.SpotInstanceRequestId
 	ui.Message(fmt.Sprintf("Waiting for spot request (%s) to become active...", *spotRequestId))
-	stateChange := StateChangeConf{
+	err = WaitUntilSpotRequestFulfilled(ctx, ec2conn, *spotRequestId)
-		Pending:   []string{"open"},
-		Target:    "active",
-		Refresh:   SpotRequestStateRefreshFunc(ec2conn, *spotRequestId),
-		StepState: state,
-	}
-	_, err = WaitForState(&stateChange)
 	if err != nil {
 		err := fmt.Errorf("Error waiting for spot request (%s) to become ready: %s", *spotRequestId, err)
 		state.Put("error", err)
@@ -227,6 +222,33 @@ func (s *StepRunSpotInstance) Run(ctx context.Context, state multistep.StateBag)
 	}
 	instanceId = *spotResp.SpotInstanceRequests[0].InstanceId
 
+	// Tag spot instance request
+	spotTags, err := s.SpotTags.EC2Tags(s.Ctx, *ec2conn.Config.Region, state)
+	if err != nil {
+		err := fmt.Errorf("Error tagging spot request: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+	spotTags.Report(ui)
+
+	if len(spotTags) > 0 && s.SpotTags.IsSet() {
+		// Retry creating tags for about 2.5 minutes
+		err = retry.Retry(0.2, 30, 11, func(_ uint) (bool, error) {
+			_, err := ec2conn.CreateTags(&ec2.CreateTagsInput{
+				Tags:      spotTags,
+				Resources: []*string{spotRequestId},
+			})
+			return true, err
+		})
+		if err != nil {
+			err := fmt.Errorf("Error tagging spot request: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
+	}
+
 	// Set the instance ID so that the cleanup works properly
 	s.instanceId = instanceId
 
@@ -344,13 +366,8 @@ func (s *StepRunSpotInstance) Cleanup(state multistep.StateBag) {
 			ui.Error(fmt.Sprintf("Error cancelling the spot request, may still be around: %s", err))
 			return
 		}
-		stateChange := StateChangeConf{
-			Pending: []string{"active", "open"},
-			Refresh: SpotRequestStateRefreshFunc(ec2conn, *s.spotRequest.SpotInstanceRequestId),
-			Target:  "cancelled",
-		}
 
-		_, err := WaitForState(&stateChange)
+		err := WaitUntilSpotRequestFulfilled(aws.BackgroundContext(), ec2conn, *s.spotRequest.SpotInstanceRequestId)
 		if err != nil {
 			ui.Error(err.Error())
 		}
@@ -364,14 +381,8 @@ func (s *StepRunSpotInstance) Cleanup(state multistep.StateBag) {
 			ui.Error(fmt.Sprintf("Error terminating instance, may still be around: %s", err))
 			return
 		}
-		stateChange := StateChangeConf{
-			Pending: []string{"pending", "running", "shutting-down", "stopped", "stopping"},
-			Refresh: InstanceStateRefreshFunc(ec2conn, s.instanceId),
-			Target:  "terminated",
-		}
 
-		_, err := WaitForState(&stateChange)
+		if err := WaitUntilInstanceTerminated(aws.BackgroundContext(), ec2conn, s.instanceId); err != nil {
-		if err != nil {
 			ui.Error(err.Error())
 		}
 	}

builder/amazon/common/step_stop_ebs_instance.go

@@ -78,9 +78,11 @@ func (s *StepStopEBSBackedInstance) Run(ctx context.Context, state multistep.Sta
 
 	// Wait for the instance to actually stop
 	ui.Say("Waiting for the instance to stop...")
-	err = ec2conn.WaitUntilInstanceStoppedWithContext(ctx, &ec2.DescribeInstancesInput{
+	err = ec2conn.WaitUntilInstanceStoppedWithContext(ctx,
-		InstanceIds: []*string{instance.InstanceId},
+		&ec2.DescribeInstancesInput{
-	})
+			InstanceIds: []*string{instance.InstanceId},
+		},
+		getWaiterOptions()...)
 
 	if err != nil {
 		err := fmt.Errorf("Error waiting for instance to stop: %s", err)

builder/amazon/ebs/builder.go

@@ -48,6 +48,7 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 				"ami_description",
 				"run_tags",
 				"run_volume_tags",
+				"spot_tags",
 				"snapshot_tags",
 				"tags",
 			},
@@ -134,6 +135,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			SourceAMI:                         b.config.SourceAmi,
 			SpotPrice:                         b.config.SpotPrice,
 			SpotPriceProduct:                  b.config.SpotPriceAutoProduct,
+			SpotTags:                          b.config.SpotTags,
 			SubnetId:                          b.config.SubnetId,
 			Tags:                              b.config.RunTags,
 			UserData:                          b.config.UserData,
@@ -189,7 +191,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			VpcId:            b.config.VpcId,
 			TemporarySGSourceCidr: b.config.TemporarySGSourceCidr,
 		},
-		&stepCleanupVolumes{
+		&awscommon.StepCleanupVolumes{
 			BlockDevices: b.config.BlockDevices,
 		},
 		instanceStep,

builder/amazon/ebs/step_create_ami.go

@@ -15,7 +15,7 @@ type stepCreateAMI struct {
 	image *ec2.Image
 }
 
-func (s *stepCreateAMI) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *stepCreateAMI) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(Config)
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	instance := state.Get("instance").(*ec2.Instance)
@@ -44,15 +44,8 @@ func (s *stepCreateAMI) Run(_ context.Context, state multistep.StateBag) multist
 	state.Put("amis", amis)
 
 	// Wait for the image to become ready
-	stateChange := awscommon.StateChangeConf{
-		Pending:   []string{"pending"},
-		Target:    "available",
-		Refresh:   awscommon.AMIStateRefreshFunc(ec2conn, *createResp.ImageId),
-		StepState: state,
-	}
-
 	ui.Say("Waiting for AMI to become ready...")
-	if _, err := awscommon.WaitForState(&stateChange); err != nil {
+	if err := awscommon.WaitUntilAMIAvailable(ctx, ec2conn, *createResp.ImageId); err != nil {
 		log.Printf("Error waiting for AMI: %s", err)
 		imagesResp, err := ec2conn.DescribeImages(&ec2.DescribeImagesInput{ImageIds: []*string{createResp.ImageId}})
 		if err != nil {

builder/amazon/ebssurrogate/builder.go

@@ -48,6 +48,7 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 				"run_tags",
 				"run_volume_tags",
 				"snapshot_tags",
+				"spot_tags",
 				"tags",
 			},
 		},
@@ -148,6 +149,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			SourceAMI:                         b.config.SourceAmi,
 			SpotPrice:                         b.config.SpotPrice,
 			SpotPriceProduct:                  b.config.SpotPriceAutoProduct,
+			SpotTags:                          b.config.SpotTags,
 			SubnetId:                          b.config.SubnetId,
 			Tags:                              b.config.RunTags,
 			UserData:                          b.config.UserData,
@@ -206,6 +208,9 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			VpcId:            b.config.VpcId,
 			TemporarySGSourceCidr: b.config.TemporarySGSourceCidr,
 		},
+		&awscommon.StepCleanupVolumes{
+			BlockDevices: b.config.BlockDevices,
+		},
 		instanceStep,
 		&awscommon.StepGetPassword{
 			Debug:     b.config.PackerDebug,

builder/amazon/ebssurrogate/step_register_ami.go

@@ -21,7 +21,7 @@ type StepRegisterAMI struct {
 	image                    *ec2.Image
 }
 
-func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepRegisterAMI) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(*Config)
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	snapshotIds := state.Get("snapshot_ids").(map[string]string)
@@ -63,15 +63,8 @@ func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multi
 	state.Put("amis", amis)
 
 	// Wait for the image to become ready
-	stateChange := awscommon.StateChangeConf{
-		Pending:   []string{"pending"},
-		Target:    "available",
-		Refresh:   awscommon.AMIStateRefreshFunc(ec2conn, *registerResp.ImageId),
-		StepState: state,
-	}
-
 	ui.Say("Waiting for AMI to become ready...")
-	if _, err := awscommon.WaitForState(&stateChange); err != nil {
+	if err := awscommon.WaitUntilAMIAvailable(ctx, ec2conn, *registerResp.ImageId); err != nil {
 		err := fmt.Errorf("Error waiting for AMI: %s", err)
 		state.Put("error", err)
 		ui.Error(err.Error())

builder/amazon/ebssurrogate/step_snapshot_volumes.go

@@ -2,7 +2,6 @@ package ebssurrogate
 
 import (
 	"context"
-	"errors"
 	"fmt"
 	"sync"
 	"time"
@@ -23,7 +22,7 @@ type StepSnapshotVolumes struct {
 	snapshotIds   map[string]string
 }
 
-func (s *StepSnapshotVolumes) snapshotVolume(deviceName string, state multistep.StateBag) error {
+func (s *StepSnapshotVolumes) snapshotVolume(ctx context.Context, deviceName string, state multistep.StateBag) error {
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	ui := state.Get("ui").(packer.Ui)
 	instance := state.Get("instance").(*ec2.Instance)
@@ -52,33 +51,12 @@ func (s *StepSnapshotVolumes) snapshotVolume(deviceName string, state multistep.
 	// Set the snapshot ID so we can delete it later
 	s.snapshotIds[deviceName] = *createSnapResp.SnapshotId
 
-	// Wait for the snapshot to be ready
+	// Wait for snapshot to be created
-	stateChange := awscommon.StateChangeConf{
+	err = awscommon.WaitUntilSnapshotDone(ctx, ec2conn, *createSnapResp.SnapshotId)
-		Pending:   []string{"pending"},
-		StepState: state,
-		Target:    "completed",
-		Refresh: func() (interface{}, string, error) {
-			resp, err := ec2conn.DescribeSnapshots(&ec2.DescribeSnapshotsInput{
-				SnapshotIds: []*string{createSnapResp.SnapshotId},
-			})
-			if err != nil {
-				return nil, "", err
-			}
-
-			if len(resp.Snapshots) == 0 {
-				return nil, "", errors.New("No snapshots found.")
-			}
-
-			s := resp.Snapshots[0]
-			return s, *s.State, nil
-		},
-	}
-
-	_, err = awscommon.WaitForState(&stateChange)
 	return err
 }
 
-func (s *StepSnapshotVolumes) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepSnapshotVolumes) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	ui := state.Get("ui").(packer.Ui)
 
 	s.snapshotIds = map[string]string{}
@@ -89,7 +67,7 @@ func (s *StepSnapshotVolumes) Run(_ context.Context, state multistep.StateBag) m
 		wg.Add(1)
 		go func(device *ec2.BlockDeviceMapping) {
 			defer wg.Done()
-			if err := s.snapshotVolume(*device.DeviceName, state); err != nil {
+			if err := s.snapshotVolume(ctx, *device.DeviceName, state); err != nil {
 				errs = multierror.Append(errs, err)
 			}
 		}(device)

builder/amazon/ebsvolume/builder.go

@@ -44,6 +44,7 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 		InterpolateFilter: &interpolate.RenderFilter{
 			Exclude: []string{
 				"run_tags",
+				"spot_tags",
 				"ebs_volumes",
 			},
 		},
@@ -132,6 +133,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			SourceAMI:                         b.config.SourceAmi,
 			SpotPrice:                         b.config.SpotPrice,
 			SpotPriceProduct:                  b.config.SpotPriceAutoProduct,
+			SpotTags:                          b.config.SpotTags,
 			SubnetId:                          b.config.SubnetId,
 			Tags:                              b.config.RunTags,
 			UserData:                          b.config.UserData,

builder/amazon/instance/builder.go

@@ -69,6 +69,7 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 				"run_volume_tags",
 				"snapshot_tags",
 				"tags",
+				"spot_tags",
 			},
 		},
 	}, configs...)
@@ -219,6 +220,7 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			SpotPriceProduct:         b.config.SpotPriceAutoProduct,
 			SubnetId:                 b.config.SubnetId,
 			Tags:                     b.config.RunTags,
+			SpotTags:                 b.config.SpotTags,
 			UserData:                 b.config.UserData,
 			UserDataFile:             b.config.UserDataFile,
 		}

builder/amazon/instance/step_register_ami.go

@@ -16,7 +16,7 @@ type StepRegisterAMI struct {
 	EnableAMISriovNetSupport bool
 }
 
-func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *StepRegisterAMI) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(*Config)
 	ec2conn := state.Get("ec2").(*ec2.EC2)
 	manifestPath := state.Get("remote_manifest_path").(string)
@@ -58,15 +58,8 @@ func (s *StepRegisterAMI) Run(_ context.Context, state multistep.StateBag) multi
 	state.Put("amis", amis)
 
 	// Wait for the image to become ready
-	stateChange := awscommon.StateChangeConf{
-		Pending:   []string{"pending"},
-		Target:    "available",
-		Refresh:   awscommon.AMIStateRefreshFunc(ec2conn, *registerResp.ImageId),
-		StepState: state,
-	}
-
 	ui.Say("Waiting for AMI to become ready...")
-	if _, err := awscommon.WaitForState(&stateChange); err != nil {
+	if err := awscommon.WaitUntilAMIAvailable(ctx, ec2conn, *registerResp.ImageId); err != nil {
 		err := fmt.Errorf("Error waiting for AMI: %s", err)
 		state.Put("error", err)
 		ui.Error(err.Error())

builder/azure/arm/artifact.go

@@ -18,6 +18,9 @@ type AdditionalDiskArtifact struct {
 }
 
 type Artifact struct {
+	// OS type: Linux, Windows
+	OSType string
+
 	// VHD
 	StorageAccountLocation string
 	OSDiskUri              string
@@ -29,20 +32,23 @@ type Artifact struct {
 	ManagedImageResourceGroupName string
 	ManagedImageName              string
 	ManagedImageLocation          string
+	ManagedImageId                string
 
 	// Additional Disks
 	AdditionalDisks *[]AdditionalDiskArtifact
 }
 
-func NewManagedImageArtifact(resourceGroup, name, location string) (*Artifact, error) {
+func NewManagedImageArtifact(osType, resourceGroup, name, location, id string) (*Artifact, error) {
 	return &Artifact{
 		ManagedImageResourceGroupName: resourceGroup,
 		ManagedImageName:              name,
 		ManagedImageLocation:          location,
+		ManagedImageId:                id,
+		OSType:                        osType,
 	}, nil
 }
 
-func NewArtifact(template *CaptureTemplate, getSasUrl func(name string) string) (*Artifact, error) {
+func NewArtifact(template *CaptureTemplate, getSasUrl func(name string) string, osType string) (*Artifact, error) {
 	if template == nil {
 		return nil, fmt.Errorf("nil capture template")
 	}
@@ -76,6 +82,7 @@ func NewArtifact(template *CaptureTemplate, getSasUrl func(name string) string)
 	}
 
 	return &Artifact{
+		OSType:                 osType,
 		OSDiskUri:              vhdUri.String(),
 		OSDiskUriReadOnlySas:   getSasUrl(getStorageUrlPath(vhdUri)),
 		TemplateUri:            templateUri.String(),
@@ -142,9 +149,11 @@ func (a *Artifact) String() string {
 	var buf bytes.Buffer
 
 	buf.WriteString(fmt.Sprintf("%s:\n\n", a.BuilderId()))
+	buf.WriteString(fmt.Sprintf("OSType: %s\n", a.OSType))
 	if a.isManagedImage() {
 		buf.WriteString(fmt.Sprintf("ManagedImageResourceGroupName: %s\n", a.ManagedImageResourceGroupName))
 		buf.WriteString(fmt.Sprintf("ManagedImageName: %s\n", a.ManagedImageName))
+		buf.WriteString(fmt.Sprintf("ManagedImageId: %s\n", a.ManagedImageId))
 		buf.WriteString(fmt.Sprintf("ManagedImageLocation: %s\n", a.ManagedImageLocation))
 	} else {
 		buf.WriteString(fmt.Sprintf("StorageAccountLocation: %s\n", a.StorageAccountLocation))

builder/azure/arm/artifact_test.go

@@ -28,7 +28,7 @@ func TestArtifactId(t *testing.T) {
 		},
 	}
 
-	artifact, err := NewArtifact(&template, getFakeSasUrl)
+	artifact, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -59,7 +59,7 @@ func TestArtifactString(t *testing.T) {
 		},
 	}
 
-	artifact, err := NewArtifact(&template, getFakeSasUrl)
+	artifact, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -80,6 +80,9 @@ func TestArtifactString(t *testing.T) {
 	if !strings.Contains(testSubject, "StorageAccountLocation: southcentralus") {
 		t.Errorf("Expected String() output to contain StorageAccountLocation")
 	}
+	if !strings.Contains(testSubject, "OSType: Linux") {
+		t.Errorf("Expected String() output to contain OSType")
+	}
 }
 
 func TestAdditionalDiskArtifactString(t *testing.T) {
@@ -107,7 +110,7 @@ func TestAdditionalDiskArtifactString(t *testing.T) {
 		},
 	}
 
-	artifact, err := NewArtifact(&template, getFakeSasUrl)
+	artifact, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -128,6 +131,9 @@ func TestAdditionalDiskArtifactString(t *testing.T) {
 	if !strings.Contains(testSubject, "StorageAccountLocation: southcentralus") {
 		t.Errorf("Expected String() output to contain StorageAccountLocation")
 	}
+	if !strings.Contains(testSubject, "OSType: Linux") {
+		t.Errorf("Expected String() output to contain OSType")
+	}
 	if !strings.Contains(testSubject, "AdditionalDiskUri (datadisk-1): https://storage.blob.core.windows.net/system/Microsoft.Compute/Images/images/packer-datadisk-1.4085bb15-3644-4641-b9cd-f575918640b4.vhd") {
 		t.Errorf("Expected String() output to contain AdditionalDiskUri")
 	}
@@ -154,7 +160,7 @@ func TestArtifactProperties(t *testing.T) {
 		},
 	}
 
-	testSubject, err := NewArtifact(&template, getFakeSasUrl)
+	testSubject, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -174,6 +180,9 @@ func TestArtifactProperties(t *testing.T) {
 	if testSubject.StorageAccountLocation != "southcentralus" {
 		t.Errorf("Expected StorageAccountLocation to be 'southcentral', but got %s", testSubject.StorageAccountLocation)
 	}
+	if testSubject.OSType != "Linux" {
+		t.Errorf("Expected OSType to be 'Linux', but got %s", testSubject.OSType)
+	}
 }
 
 func TestAdditionalDiskArtifactProperties(t *testing.T) {
@@ -201,7 +210,7 @@ func TestAdditionalDiskArtifactProperties(t *testing.T) {
 		},
 	}
 
-	testSubject, err := NewArtifact(&template, getFakeSasUrl)
+	testSubject, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -221,6 +230,9 @@ func TestAdditionalDiskArtifactProperties(t *testing.T) {
 	if testSubject.StorageAccountLocation != "southcentralus" {
 		t.Errorf("Expected StorageAccountLocation to be 'southcentral', but got %s", testSubject.StorageAccountLocation)
 	}
+	if testSubject.OSType != "Linux" {
+		t.Errorf("Expected OSType to be 'Linux', but got %s", testSubject.OSType)
+	}
 	if testSubject.AdditionalDisks == nil {
 		t.Errorf("Expected AdditionalDisks to be not nil")
 	}
@@ -253,7 +265,7 @@ func TestArtifactOverHyphenatedCaptureUri(t *testing.T) {
 		},
 	}
 
-	testSubject, err := NewArtifact(&template, getFakeSasUrl)
+	testSubject, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err != nil {
 		t.Fatalf("err=%s", err)
 	}
@@ -266,7 +278,7 @@ func TestArtifactOverHyphenatedCaptureUri(t *testing.T) {
 func TestArtifactRejectMalformedTemplates(t *testing.T) {
 	template := CaptureTemplate{}
 
-	_, err := NewArtifact(&template, getFakeSasUrl)
+	_, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err == nil {
 		t.Fatalf("Expected artifact creation to fail, but it succeeded.")
 	}
@@ -289,7 +301,7 @@ func TestArtifactRejectMalformedStorageUri(t *testing.T) {
 		},
 	}
 
-	_, err := NewArtifact(&template, getFakeSasUrl)
+	_, err := NewArtifact(&template, getFakeSasUrl, "Linux")
 	if err == nil {
 		t.Fatalf("Expected artifact creation to fail, but it succeeded.")
 	}

builder/azure/arm/builder.go

@@ -255,7 +255,8 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 	}
 
 	if b.config.isManagedImage() {
-		return NewManagedImageArtifact(b.config.ManagedImageResourceGroupName, b.config.ManagedImageName, b.config.manageImageLocation)
+		managedImageID := fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/images/%s", b.config.SubscriptionID, b.config.ManagedImageResourceGroupName, b.config.ManagedImageName)
+		return NewManagedImageArtifact(b.config.OSType, b.config.ManagedImageResourceGroupName, b.config.ManagedImageName, b.config.manageImageLocation, managedImageID)
 	} else if template, ok := b.stateBag.GetOk(constants.ArmCaptureTemplate); ok {
 		return NewArtifact(
 			template.(*CaptureTemplate),
@@ -266,7 +267,8 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 				options.Expiry = time.Now().AddDate(0, 1, 0).UTC() // one month
 				sasUrl, _ := blob.GetSASURI(options)
 				return sasUrl
-			})
+			},
+			b.config.OSType)
 	}
 
 	return &Artifact{}, nil

builder/azure/arm/config.go

@@ -47,7 +47,7 @@ const (
 	//  -> ^[^_\W][\w-._]{0,79}(?<![-.])$
 	//
 	// This is not an exhaustive match, but it should be extremely close.
-	validResourceGroupNameRe = "^[^_\\W][\\w-._\\(\\)]{0,63}$"
+	validResourceGroupNameRe = "^[^_\\W][\\w-._\\(\\)]{0,89}$"
 	validManagedDiskName     = "^[^_\\W][\\w-._)]{0,79}$"
 )
 
@@ -150,7 +150,7 @@ type Config struct {
 	winrmCertificate string
 
 	Comm communicator.Config `mapstructure:",squash"`
-	ctx  *interpolate.Context
+	ctx  interpolate.Context
 
 	//Cleanup
 	AsyncResourceGroupDelete bool `mapstructure:"async_resourcegroup_delete"`
@@ -258,10 +258,10 @@ func (c *Config) createCertificate() (string, error) {
 
 func newConfig(raws ...interface{}) (*Config, []string, error) {
 	var c Config
-
+	c.ctx.Funcs = TemplateFuncs
 	err := config.Decode(&c, &config.DecodeOpts{
 		Interpolate:        true,
-		InterpolateContext: c.ctx,
+		InterpolateContext: &c.ctx,
 	}, raws...)
 
 	if err != nil {
@@ -299,7 +299,7 @@ func newConfig(raws ...interface{}) (*Config, []string, error) {
 	}
 
 	var errs *packer.MultiError
-	errs = packer.MultiErrorAppend(errs, c.Comm.Prepare(c.ctx)...)
+	errs = packer.MultiErrorAppend(errs, c.Comm.Prepare(&c.ctx)...)
 
 	assertRequiredParametersSet(&c, errs)
 	assertTagProperties(&c, errs)

builder/azure/arm/template_funcs.go

@@ -0,0 +1,43 @@
+package arm
+
+import (
+	"bytes"
+	"text/template"
+)
+
+func isValidByteValue(b byte) bool {
+	if '0' <= b && b <= '9' {
+		return true
+	}
+	if 'a' <= b && b <= 'z' {
+		return true
+	}
+	if 'A' <= b && b <= 'Z' {
+		return true
+	}
+	return b == '.' || b == '_' || b == '-'
+}
+
+// Clean up image name by replacing invalid characters with "-"
+// Names are not allowed to end in '.', '-', or  '_' and are trimmed.
+func templateCleanImageName(s string) string {
+	if ok, _ := assertManagedImageName(s, ""); ok {
+		return s
+	}
+	b := []byte(s)
+	newb := make([]byte, len(b))
+	for i := range newb {
+		if isValidByteValue(b[i]) {
+			newb[i] = b[i]
+		} else {
+			newb[i] = '-'
+		}
+	}
+
+	newb = bytes.TrimRight(newb, "-_.")
+	return string(newb)
+}
+
+var TemplateFuncs = template.FuncMap{
+	"clean_image_name": templateCleanImageName,
+}

builder/azure/arm/template_funcs_test.go

@@ -0,0 +1,49 @@
+package arm
+
+import "testing"
+
+func TestTemplateCleanImageName(t *testing.T) {
+	vals := []struct {
+		origName string
+		expected string
+	}{
+		// test that valid name is unchanged
+		{
+			origName: "abcde-012345xyz",
+			expected: "abcde-012345xyz",
+		},
+		// test that colons are converted to hyphens
+		{
+			origName: "abcde-012345v1.0:0",
+			expected: "abcde-012345v1.0-0",
+		},
+		// Name starting with number is not valid, but not in scope of this
+		// function to correct
+		{
+			origName: "012345v1.0:0",
+			expected: "012345v1.0-0",
+		},
+		// Name over 80 chars is not valid, but not corrected by this function.
+		{
+			origName: "l012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789",
+			expected: "l012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789",
+		},
+		// Name cannot end in a -Name over 80 chars is not valid, but not corrected by this function.
+		{
+			origName: "abcde-:_",
+			expected: "abcde",
+		},
+		// Lost of special characters
+		{
+			origName: "My()./-_:&^ $%[]#'@name",
+			expected: "My--.--_-----------name",
+		},
+	}
+
+	for _, v := range vals {
+		name := templateCleanImageName(v.origName)
+		if name != v.expected {
+			t.Fatalf("template names do not match: expected %s got %s\n", v.expected, name)
+		}
+	}
+}

builder/azure/arm/tempname.go

@@ -2,13 +2,19 @@ package arm
 
 import (
 	"fmt"
+	"strings"
 
 	"github.com/hashicorp/packer/builder/azure/common"
 )
 
 const (
-	TempNameAlphabet     = "0123456789bcdfghjklmnpqrstvwxyz"
+	TempNameAlphabet = "0123456789bcdfghjklmnpqrstvwxyz"
-	TempPasswordAlphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
+	numbers   = "0123456789"
+	lowerCase = "abcdefghijklmnopqrstuvwxyz"
+	upperCase = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+
+	TempPasswordAlphabet = numbers + lowerCase + upperCase
 )
 
 type TempName struct {
@@ -39,8 +45,37 @@ func NewTempName() *TempName {
 	tempName.VirtualNetworkName = fmt.Sprintf("pkrvn%s", suffix)
 	tempName.ResourceGroupName = fmt.Sprintf("packer-Resource-Group-%s", suffix)
 
-	tempName.AdminPassword = common.RandomString(TempPasswordAlphabet, 32)
+	tempName.AdminPassword = generatePassword()
 	tempName.CertificatePassword = common.RandomString(TempPasswordAlphabet, 32)
 
 	return tempName
 }
+
+// generate a password that is acceptable to Azure
+// Three of the four items must be met.
+//  1. Contains an uppercase character
+//  2. Contains a lowercase character
+//  3. Contains a numeric digit
+//  4. Contains a special character
+func generatePassword() string {
+	var s string
+	for i := 0; i < 100; i++ {
+		s := common.RandomString(TempPasswordAlphabet, 32)
+		if !strings.ContainsAny(s, numbers) {
+			continue
+		}
+
+		if !strings.ContainsAny(s, lowerCase) {
+			continue
+		}
+
+		if !strings.ContainsAny(s, upperCase) {
+			continue
+		}
+
+		return s
+	}
+
+	// if an acceptable password cannot be generated in 100 tries, give up
+	return s
+}

builder/azure/arm/tempname_test.go

@@ -41,6 +41,20 @@ func TestTempNameShouldCreatePrefixedRandomNames(t *testing.T) {
 	}
 }
 
+func TestTempAdminPassword(t *testing.T) {
+	tempName := NewTempName()
+
+	if !strings.ContainsAny(tempName.AdminPassword, numbers) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", numbers)
+	}
+	if !strings.ContainsAny(tempName.AdminPassword, lowerCase) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", lowerCase)
+	}
+	if !strings.ContainsAny(tempName.AdminPassword, upperCase) {
+		t.Errorf("Expected AdminPassword to contain at least one of '%s'!", upperCase)
+	}
+}
+
 func TestTempNameShouldHaveSameSuffix(t *testing.T) {
 	tempName := NewTempName()
 	suffix := tempName.ComputeName[5:]

builder/cloudstack/config.go

@@ -27,26 +27,27 @@ type Config struct {
 	HTTPGetOnly  bool          `mapstructure:"http_get_only"`
 	SSLNoVerify  bool          `mapstructure:"ssl_no_verify"`
 
-	CIDRList             []string `mapstructure:"cidr_list"`
+	CIDRList               []string `mapstructure:"cidr_list"`
-	CreateSecurityGroup  bool     `mapstructure:"create_security_group"`
+	CreateSecurityGroup    bool     `mapstructure:"create_security_group"`
-	DiskOffering         string   `mapstructure:"disk_offering"`
+	DiskOffering           string   `mapstructure:"disk_offering"`
-	DiskSize             int64    `mapstructure:"disk_size"`
+	DiskSize               int64    `mapstructure:"disk_size"`
-	Expunge              bool     `mapstructure:"expunge"`
+	Expunge                bool     `mapstructure:"expunge"`
-	Hypervisor           string   `mapstructure:"hypervisor"`
+	Hypervisor             string   `mapstructure:"hypervisor"`
-	InstanceName         string   `mapstructure:"instance_name"`
+	InstanceName           string   `mapstructure:"instance_name"`
-	Keypair              string   `mapstructure:"keypair"`
+	Keypair                string   `mapstructure:"keypair"`
-	Network              string   `mapstructure:"network"`
+	Network                string   `mapstructure:"network"`
-	Project              string   `mapstructure:"project"`
+	Project                string   `mapstructure:"project"`
-	PublicIPAddress      string   `mapstructure:"public_ip_address"`
+	PublicIPAddress        string   `mapstructure:"public_ip_address"`
-	SecurityGroups       []string `mapstructure:"security_groups"`
+	SecurityGroups         []string `mapstructure:"security_groups"`
-	ServiceOffering      string   `mapstructure:"service_offering"`
+	ServiceOffering        string   `mapstructure:"service_offering"`
-	SourceISO            string   `mapstructure:"source_iso"`
+	PreventFirewallChanges bool     `mapstructure:"prevent_firewall_changes"`
-	SourceTemplate       string   `mapstructure:"source_template"`
+	SourceISO              string   `mapstructure:"source_iso"`
-	TemporaryKeypairName string   `mapstructure:"temporary_keypair_name"`
+	SourceTemplate         string   `mapstructure:"source_template"`
-	UseLocalIPAddress    bool     `mapstructure:"use_local_ip_address"`
+	TemporaryKeypairName   string   `mapstructure:"temporary_keypair_name"`
-	UserData             string   `mapstructure:"user_data"`
+	UseLocalIPAddress      bool     `mapstructure:"use_local_ip_address"`
-	UserDataFile         string   `mapstructure:"user_data_file"`
+	UserData               string   `mapstructure:"user_data"`
-	Zone                 string   `mapstructure:"zone"`
+	UserDataFile           string   `mapstructure:"user_data_file"`
+	Zone                   string   `mapstructure:"zone"`
 
 	TemplateName            string `mapstructure:"template_name"`
 	TemplateDisplayText     string `mapstructure:"template_display_text"`

builder/cloudstack/step_configure_networking.go

@@ -117,6 +117,11 @@ func (s *stepSetupNetworking) Run(_ context.Context, state multistep.StateBag) m
 	// Store the port forward ID.
 	state.Put("port_forward_id", forward.Id)
 
+	if config.PreventFirewallChanges {
+		ui.Message("Networking has been setup (without firewall changes)!")
+		return multistep.ActionContinue
+	}
+
 	if network.Vpcid != "" {
 		ui.Message("Creating network ACL rule...")
 

builder/cloudstack/step_create_instance.go

@@ -47,7 +47,9 @@ func (s *stepCreateInstance) Run(_ context.Context, state multistep.StateBag) mu
 	p.SetDisplayname("Created by Packer")
 
 	if keypair, ok := state.GetOk("keypair"); ok {
-		p.SetKeypair(keypair.(string))
+		kp := keypair.(string)
+		ui.Message(fmt.Sprintf("Using keypair: %s", kp))
+		p.SetKeypair(kp)
 	}
 
 	if securitygroups, ok := state.GetOk("security_groups"); ok {
@@ -120,6 +122,7 @@ func (s *stepCreateInstance) Run(_ context.Context, state multistep.StateBag) mu
 	}
 
 	ui.Message("Instance has been created!")
+	ui.Message(fmt.Sprintf("Instance ID: %s", instance.Id))
 
 	// In debug-mode, we output the password
 	if s.Debug {

builder/cloudstack/step_create_template.go

@@ -51,7 +51,7 @@ func (s *stepCreateTemplate) Run(_ context.Context, state multistep.StateBag) mu
 	}
 
 	ui.Message("Retrieving the ROOT volume ID...")
-	volumeID, err := getRootVolumeID(client, instanceID)
+	volumeID, err := getRootVolumeID(client, instanceID, config.Project)
 	if err != nil {
 		state.Put("error", err)
 		ui.Error(err.Error())
@@ -89,13 +89,16 @@ func (s *stepCreateTemplate) Cleanup(state multistep.StateBag) {
 	// Nothing to cleanup for this step.
 }
 
-func getRootVolumeID(client *cloudstack.CloudStackClient, instanceID string) (string, error) {
+func getRootVolumeID(client *cloudstack.CloudStackClient, instanceID, projectID string) (string, error) {
 	// Retrieve the virtual machine object.
 	p := client.Volume.NewListVolumesParams()
 
 	// Set the type and virtual machine ID
 	p.SetType("ROOT")
 	p.SetVirtualmachineid(instanceID)
+	if projectID != "" {
+		p.SetProjectid(projectID)
+	}
 
 	volumes, err := client.Volume.ListVolumes(p)
 	if err != nil {

builder/cloudstack/step_keypair.go

@@ -60,6 +60,12 @@ func (s *stepKeypair) Run(_ context.Context, state multistep.StateBag) multistep
 	ui.Say(fmt.Sprintf("Creating temporary keypair: %s ...", s.TemporaryKeyPairName))
 
 	p := client.SSH.NewCreateSSHKeyPairParams(s.TemporaryKeyPairName)
+
+	cfg := state.Get("config").(*Config)
+	if cfg.Project != "" {
+		p.SetProjectid(cfg.Project)
+	}
+
 	keypair, err := client.SSH.CreateSSHKeyPair(p)
 	if err != nil {
 		err := fmt.Errorf("Error creating temporary keypair: %s", err)
@@ -120,12 +126,16 @@ func (s *stepKeypair) Cleanup(state multistep.StateBag) {
 
 	ui := state.Get("ui").(packer.Ui)
 	client := state.Get("client").(*cloudstack.CloudStackClient)
+	cfg := state.Get("config").(*Config)
+
+	p := client.SSH.NewDeleteSSHKeyPairParams(s.TemporaryKeyPairName)
+	if cfg.Project != "" {
+		p.SetProjectid(cfg.Project)
+	}
 
 	ui.Say(fmt.Sprintf("Deleting temporary keypair: %s ...", s.TemporaryKeyPairName))
 
-	_, err := client.SSH.DeleteSSHKeyPair(client.SSH.NewDeleteSSHKeyPairParams(
+	_, err := client.SSH.DeleteSSHKeyPair(p)
-		s.TemporaryKeyPairName,
-	))
 	if err != nil {
 		ui.Error(err.Error())
 		ui.Error(fmt.Sprintf(

builder/digitalocean/config.go

@@ -4,6 +4,7 @@ import (
 	"errors"
 	"fmt"
 	"os"
+	"regexp"
 	"time"
 
 	"github.com/hashicorp/packer/common"
@@ -35,6 +36,7 @@ type Config struct {
 	DropletName       string        `mapstructure:"droplet_name"`
 	UserData          string        `mapstructure:"user_data"`
 	UserDataFile      string        `mapstructure:"user_data_file"`
+	Tags              []string      `mapstructure:"tags"`
 
 	ctx interpolate.Context
 }
@@ -121,6 +123,17 @@ func NewConfig(raws ...interface{}) (*Config, []string, error) {
 		}
 	}
 
+	if c.Tags == nil {
+		c.Tags = make([]string, 0)
+	}
+	tagRe := regexp.MustCompile("^[[:alnum:]:_-]{1,255}$")
+
+	for _, t := range c.Tags {
+		if !tagRe.MatchString(t) {
+			errs = packer.MultiErrorAppend(errs, errors.New(fmt.Sprintf("invalid tag: %s", t)))
+		}
+	}
+
 	if errs != nil && len(errs.Errors) > 0 {
 		return nil, nil, errs
 	}

builder/digitalocean/step_create_droplet.go

@@ -49,6 +49,7 @@ func (s *stepCreateDroplet) Run(_ context.Context, state multistep.StateBag) mul
 		Monitoring:        c.Monitoring,
 		IPv6:              c.IPv6,
 		UserData:          userData,
+		Tags:              c.Tags,
 	})
 	if err != nil {
 		err := fmt.Errorf("Error creating droplet: %s", err)

builder/lxc/command.go

@@ -1,7 +1,11 @@
 package lxc
 
 import (
+	"bytes"
+	"fmt"
+	"log"
 	"os/exec"
+	"strings"
 )
 
 // CommandWrapper is a type that given a command, will possibly modify that
@@ -13,3 +17,25 @@ type CommandWrapper func(string) (string, error)
 func ShellCommand(command string) *exec.Cmd {
 	return exec.Command("/bin/sh", "-c", command)
 }
+
+func RunCommand(args ...string) error {
+	var stdout, stderr bytes.Buffer
+
+	log.Printf("Executing args: %#v", args)
+	cmd := exec.Command(args[0], args[1:]...)
+	cmd.Stdout = &stdout
+	cmd.Stderr = &stderr
+	err := cmd.Run()
+
+	stdoutString := strings.TrimSpace(stdout.String())
+	stderrString := strings.TrimSpace(stderr.String())
+
+	if _, ok := err.(*exec.ExitError); ok {
+		err = fmt.Errorf("Command error: %s", stderrString)
+	}
+
+	log.Printf("stdout: %s", stdoutString)
+	log.Printf("stderr: %s", stderrString)
+
+	return err
+}

builder/lxc/communicator.go

@@ -59,7 +59,6 @@ func (c *LxcAttachCommunicator) Start(cmd *packer.RemoteCmd) error {
 }
 
 func (c *LxcAttachCommunicator) Upload(dst string, r io.Reader, fi *os.FileInfo) error {
-	dst = filepath.Join(c.RootFs, dst)
 	log.Printf("Uploading to rootfs: %s", dst)
 	tf, err := ioutil.TempFile("", "packer-lxc-attach")
 	if err != nil {
@@ -68,7 +67,11 @@ func (c *LxcAttachCommunicator) Upload(dst string, r io.Reader, fi *os.FileInfo)
 	defer os.Remove(tf.Name())
 	io.Copy(tf, r)
 
-	cpCmd, err := c.CmdWrapper(fmt.Sprintf("sudo cp %s %s", tf.Name(), dst))
+	attachCommand := []string{"cat", "%s", " | ", "lxc-attach"}
+	attachCommand = append(attachCommand, c.AttachOptions...)
+	attachCommand = append(attachCommand, []string{"--name", "%s", "--", "/bin/sh -c \"/bin/cat > %s\""}...)
+
+	cpCmd, err := c.CmdWrapper(fmt.Sprintf(strings.Join(attachCommand, " "), tf.Name(), c.ContainerName, dst))
 	if err != nil {
 		return err
 	}
@@ -78,14 +81,14 @@ func (c *LxcAttachCommunicator) Upload(dst string, r io.Reader, fi *os.FileInfo)
 		// rename tempfile to match original file name. This makes sure that if file is being
 		// moved into a directory, the filename is preserved instead of a temp name.
 		adjustedTempName := filepath.Join(tfDir, (*fi).Name())
-		mvCmd, err := c.CmdWrapper(fmt.Sprintf("sudo mv %s %s", tf.Name(), adjustedTempName))
+		mvCmd, err := c.CmdWrapper(fmt.Sprintf("mv %s %s", tf.Name(), adjustedTempName))
 		if err != nil {
 			return err
 		}
 		defer os.Remove(adjustedTempName)
 		ShellCommand(mvCmd).Run()
 		// change cpCmd to use new file name as source
-		cpCmd, err = c.CmdWrapper(fmt.Sprintf("sudo cp %s %s", adjustedTempName, dst))
+		cpCmd, err = c.CmdWrapper(fmt.Sprintf(strings.Join(attachCommand, " "), adjustedTempName, c.ContainerName, dst))
 		if err != nil {
 			return err
 		}
@@ -100,7 +103,7 @@ func (c *LxcAttachCommunicator) UploadDir(dst string, src string, exclude []stri
 	// TODO: remove any file copied if it appears in `exclude`
 	dest := filepath.Join(c.RootFs, dst)
 	log.Printf("Uploading directory '%s' to rootfs '%s'", src, dest)
-	cpCmd, err := c.CmdWrapper(fmt.Sprintf("sudo cp -R %s/. %s", src, dest))
+	cpCmd, err := c.CmdWrapper(fmt.Sprintf("cp -R %s/. %s", src, dest))
 	if err != nil {
 		return err
 	}
@@ -131,7 +134,7 @@ func (c *LxcAttachCommunicator) DownloadDir(src string, dst string, exclude []st
 func (c *LxcAttachCommunicator) Execute(commandString string) (*exec.Cmd, error) {
 	log.Printf("Executing with lxc-attach in container: %s %s %s", c.ContainerName, c.RootFs, commandString)
 
-	attachCommand := []string{"sudo", "lxc-attach"}
+	attachCommand := []string{"lxc-attach"}
 	attachCommand = append(attachCommand, c.AttachOptions...)
 	attachCommand = append(attachCommand, []string{"--name", "%s", "--", "/bin/sh -c \"%s\""}...)
 

builder/lxc/step_export.go

@@ -1,15 +1,13 @@
 package lxc
 
 import (
-	"bytes"
 	"context"
 	"fmt"
 	"io"
 	"log"
 	"os"
-	"os/exec"
+	"os/user"
 	"path/filepath"
-	"strings"
 
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
@@ -23,7 +21,16 @@ func (s *stepExport) Run(_ context.Context, state multistep.StateBag) multistep.
 
 	name := config.ContainerName
 
-	containerDir := fmt.Sprintf("/var/lib/lxc/%s", name)
+	lxc_dir := "/var/lib/lxc"
+	user, err := user.Current()
+	if err != nil {
+		log.Print("Cannot find current user. Falling back to /var/lib/lxc...")
+	}
+	if user.Uid != "0" && user.HomeDir != "" {
+		lxc_dir = filepath.Join(user.HomeDir, ".local", "share", "lxc")
+	}
+
+	containerDir := filepath.Join(lxc_dir, name)
 	outputPath := filepath.Join(config.OutputDir, "rootfs.tar.gz")
 	configFilePath := filepath.Join(config.OutputDir, "lxc-config")
 
@@ -47,7 +54,7 @@ func (s *stepExport) Run(_ context.Context, state multistep.StateBag) multistep.
 
 	_, err = io.Copy(configFile, originalConfigFile)
 
-	commands := make([][]string, 4)
+	commands := make([][]string, 3)
 	commands[0] = []string{
 		"lxc-stop", "--name", name,
 	}
@@ -57,13 +64,10 @@ func (s *stepExport) Run(_ context.Context, state multistep.StateBag) multistep.
 	commands[2] = []string{
 		"chmod", "+x", configFilePath,
 	}
-	commands[3] = []string{
-		"sh", "-c", "chown $USER:`id -gn` " + filepath.Join(config.OutputDir, "*"),
-	}
 
 	ui.Say("Exporting container...")
 	for _, command := range commands {
-		err := s.SudoCommand(command...)
+		err := RunCommand(command...)
 		if err != nil {
 			err := fmt.Errorf("Error exporting container: %s", err)
 			state.Put("error", err)
@@ -76,25 +80,3 @@ func (s *stepExport) Run(_ context.Context, state multistep.StateBag) multistep.
 }
 
 func (s *stepExport) Cleanup(state multistep.StateBag) {}
-
-func (s *stepExport) SudoCommand(args ...string) error {
-	var stdout, stderr bytes.Buffer
-
-	log.Printf("Executing sudo command: %#v", args)
-	cmd := exec.Command("sudo", args...)
-	cmd.Stdout = &stdout
-	cmd.Stderr = &stderr
-	err := cmd.Run()
-
-	stdoutString := strings.TrimSpace(stdout.String())
-	stderrString := strings.TrimSpace(stderr.String())
-
-	if _, ok := err.(*exec.ExitError); ok {
-		err = fmt.Errorf("Sudo command error: %s", stderrString)
-	}
-
-	log.Printf("stdout: %s", stdoutString)
-	log.Printf("stderr: %s", stderrString)
-
-	return err
-}

builder/lxc/step_lxc_create.go

@@ -1,13 +1,11 @@
 package lxc
 
 import (
-	"bytes"
 	"context"
 	"fmt"
 	"log"
-	"os/exec"
+	"os/user"
 	"path/filepath"
-	"strings"
 
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
@@ -23,6 +21,13 @@ func (s *stepLxcCreate) Run(_ context.Context, state multistep.StateBag) multist
 
 	// TODO: read from env
 	lxc_dir := "/var/lib/lxc"
+	user, err := user.Current()
+	if err != nil {
+		log.Print("Cannot find current user. Falling back to /var/lib/lxc...")
+	}
+	if user.Uid != "0" && user.HomeDir != "" {
+		lxc_dir = filepath.Join(user.HomeDir, ".local", "share", "lxc")
+	}
 	rootfs := filepath.Join(lxc_dir, name, "rootfs")
 
 	if config.PackerForce {
@@ -30,7 +35,9 @@ func (s *stepLxcCreate) Run(_ context.Context, state multistep.StateBag) multist
 	}
 
 	commands := make([][]string, 3)
-	commands[0] = append(config.EnvVars, "lxc-create")
+	commands[0] = append(commands[0], "env")
+	commands[0] = append(commands[0], config.EnvVars...)
+	commands[0] = append(commands[0], "lxc-create")
 	commands[0] = append(commands[0], config.CreateOptions...)
 	commands[0] = append(commands[0], []string{"-n", name, "-t", config.Name, "--"}...)
 	commands[0] = append(commands[0], config.Parameters...)
@@ -42,8 +49,7 @@ func (s *stepLxcCreate) Run(_ context.Context, state multistep.StateBag) multist
 
 	ui.Say("Creating container...")
 	for _, command := range commands {
-		log.Printf("Executing sudo command: %#v", command)
+		err := RunCommand(command...)
-		err := s.SudoCommand(command...)
 		if err != nil {
 			err := fmt.Errorf("Error creating container: %s", err)
 			state.Put("error", err)
@@ -66,29 +72,7 @@ func (s *stepLxcCreate) Cleanup(state multistep.StateBag) {
 	}
 
 	ui.Say("Unregistering and deleting virtual machine...")
-	if err := s.SudoCommand(command...); err != nil {
+	if err := RunCommand(command...); err != nil {
 		ui.Error(fmt.Sprintf("Error deleting virtual machine: %s", err))
 	}
 }
-
-func (s *stepLxcCreate) SudoCommand(args ...string) error {
-	var stdout, stderr bytes.Buffer
-
-	log.Printf("Executing sudo command: %#v", args)
-	cmd := exec.Command("sudo", args...)
-	cmd.Stdout = &stdout
-	cmd.Stderr = &stderr
-	err := cmd.Run()
-
-	stdoutString := strings.TrimSpace(stdout.String())
-	stderrString := strings.TrimSpace(stderr.String())
-
-	if _, ok := err.(*exec.ExitError); ok {
-		err = fmt.Errorf("Sudo command error: %s", stderrString)
-	}
-
-	log.Printf("stdout: %s", stdoutString)
-	log.Printf("stderr: %s", stderrString)
-
-	return err
-}

builder/lxd/step_lxd_launch.go

@@ -26,7 +26,7 @@ func (s *stepLxdLaunch) Run(_ context.Context, state multistep.StateBag) multist
 	}
 
 	for k, v := range config.LaunchConfig {
-		launch_args = append(launch_args, fmt.Sprintf("--config %s=%s", k, v))
+		launch_args = append(launch_args, "--config", fmt.Sprintf("%s=%s", k, v))
 	}
 
 	ui.Say("Creating container...")

builder/openstack/access_config.go

@@ -194,6 +194,20 @@ func (c *AccessConfig) imageV2Client() (*gophercloud.ServiceClient, error) {
 	})
 }
 
+func (c *AccessConfig) blockStorageV3Client() (*gophercloud.ServiceClient, error) {
+	return openstack.NewBlockStorageV3(c.osClient, gophercloud.EndpointOpts{
+		Region:       c.Region,
+		Availability: c.getEndpointType(),
+	})
+}
+
+func (c *AccessConfig) networkV2Client() (*gophercloud.ServiceClient, error) {
+	return openstack.NewNetworkV2(c.osClient, gophercloud.EndpointOpts{
+		Region:       c.Region,
+		Availability: c.getEndpointType(),
+	})
+}
+
 func (c *AccessConfig) getEndpointType() gophercloud.Availability {
 	if c.EndpointType == "internal" || c.EndpointType == "internalURL" {
 		return gophercloud.AvailabilityInternal

builder/openstack/builder.go

@@ -86,17 +86,26 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			PrivateKeyFile:       b.config.RunConfig.Comm.SSHPrivateKey,
 			SSHAgentAuth:         b.config.RunConfig.Comm.SSHAgentAuth,
 		},
+		&StepCreateVolume{
+			UseBlockStorageVolume:  b.config.UseBlockStorageVolume,
+			SourceImage:            b.config.SourceImage,
+			VolumeName:             b.config.VolumeName,
+			VolumeType:             b.config.VolumeType,
+			VolumeAvailabilityZone: b.config.VolumeAvailabilityZone,
+		},
 		&StepRunSourceServer{
-			Name:             b.config.InstanceName,
+			Name:                  b.config.InstanceName,
-			SourceImage:      b.config.SourceImage,
+			SourceImage:           b.config.SourceImage,
-			SourceImageName:  b.config.SourceImageName,
+			SourceImageName:       b.config.SourceImageName,
-			SecurityGroups:   b.config.SecurityGroups,
+			SecurityGroups:        b.config.SecurityGroups,
-			Networks:         b.config.Networks,
+			Networks:              b.config.Networks,
-			AvailabilityZone: b.config.AvailabilityZone,
+			Ports:                 b.config.Ports,
-			UserData:         b.config.UserData,
+			AvailabilityZone:      b.config.AvailabilityZone,
-			UserDataFile:     b.config.UserDataFile,
+			UserData:              b.config.UserData,
-			ConfigDrive:      b.config.ConfigDrive,
+			UserDataFile:          b.config.UserDataFile,
-			InstanceMetadata: b.config.InstanceMetadata,
+			ConfigDrive:           b.config.ConfigDrive,
+			InstanceMetadata:      b.config.InstanceMetadata,
+			UseBlockStorageVolume: b.config.UseBlockStorageVolume,
 		},
 		&StepGetPassword{
 			Debug: b.config.PackerDebug,
@@ -106,9 +115,9 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 			Wait: b.config.RackconnectWait,
 		},
 		&StepAllocateIp{
-			FloatingIpPool: b.config.FloatingIpPool,
+			FloatingIPNetwork: b.config.FloatingIPNetwork,
-			FloatingIp:     b.config.FloatingIp,
+			FloatingIP:        b.config.FloatingIP,
-			ReuseIps:       b.config.ReuseIps,
+			ReuseIPs:          b.config.ReuseIPs,
 		},
 		&communicator.StepConnect{
 			Config: &b.config.RunConfig.Comm,
@@ -123,7 +132,12 @@ func (b *Builder) Run(ui packer.Ui, hook packer.Hook, cache packer.Cache) (packe
 		},
 		&common.StepProvision{},
 		&StepStopServer{},
-		&stepCreateImage{},
+		&StepDetachVolume{
+			UseBlockStorageVolume: b.config.UseBlockStorageVolume,
+		},
+		&stepCreateImage{
+			UseBlockStorageVolume: b.config.UseBlockStorageVolume,
+		},
 		&stepUpdateImageVisibility{},
 		&stepAddImageMembers{},
 	}

builder/openstack/networks.go

@@ -0,0 +1,125 @@
+package openstack
+
+import (
+	"fmt"
+
+	"github.com/google/uuid"
+	"github.com/gophercloud/gophercloud"
+	"github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/attachinterfaces"
+	"github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/external"
+	"github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/layer3/floatingips"
+	"github.com/gophercloud/gophercloud/openstack/networking/v2/networks"
+	"github.com/gophercloud/gophercloud/pagination"
+)
+
+// CheckFloatingIP gets a floating IP by its ID and checks if it is already
+// associated with any internal interface.
+// It returns floating IP if it can be used.
+func CheckFloatingIP(client *gophercloud.ServiceClient, id string) (*floatingips.FloatingIP, error) {
+	floatingIP, err := floatingips.Get(client, id).Extract()
+	if err != nil {
+		return nil, err
+	}
+	if floatingIP.PortID != "" {
+		return nil, fmt.Errorf("provided floating IP '%s' is already associated with port '%s'",
+			id, floatingIP.PortID)
+	}
+
+	return floatingIP, nil
+}
+
+// FindFreeFloatingIP returns free unassociated floating IP.
+// It will return first floating IP if there are many.
+func FindFreeFloatingIP(client *gophercloud.ServiceClient) (*floatingips.FloatingIP, error) {
+	var freeFloatingIP *floatingips.FloatingIP
+
+	pager := floatingips.List(client, floatingips.ListOpts{
+		Status: "DOWN",
+	})
+	err := pager.EachPage(func(page pagination.Page) (bool, error) {
+		candidates, err := floatingips.ExtractFloatingIPs(page)
+		if err != nil {
+			return false, err // stop and throw error out
+		}
+
+		for _, candidate := range candidates {
+			if candidate.PortID != "" {
+				continue // this floating IP is associated with port, move to next in list
+			}
+
+			// Floating IP is able to be allocated.
+			freeFloatingIP = &candidate
+			return false, nil // stop iterating over pages
+		}
+		return true, nil // try the next page
+	})
+	if err != nil {
+		return nil, err
+	}
+	if freeFloatingIP == nil {
+		return nil, fmt.Errorf("no free floating IPs found")
+	}
+
+	return freeFloatingIP, nil
+}
+
+// GetInstancePortID returns internal port of the instance that can be used for
+// the association of a floating IP.
+// It will return an ID of a first port if there are many.
+func GetInstancePortID(client *gophercloud.ServiceClient, id string) (string, error) {
+	interfacesPage, err := attachinterfaces.List(client, id).AllPages()
+	if err != nil {
+		return "", err
+	}
+	interfaces, err := attachinterfaces.ExtractInterfaces(interfacesPage)
+	if err != nil {
+		return "", err
+	}
+	if len(interfaces) == 0 {
+		return "", fmt.Errorf("instance '%s' has no interfaces", id)
+	}
+
+	return interfaces[0].PortID, nil
+}
+
+// CheckFloatingIPNetwork checks provided network reference and returns a valid
+// Networking service ID.
+func CheckFloatingIPNetwork(client *gophercloud.ServiceClient, networkRef string) (string, error) {
+	if _, err := uuid.Parse(networkRef); err != nil {
+		return GetFloatingIPNetworkIDByName(client, networkRef)
+	}
+
+	return networkRef, nil
+}
+
+// ExternalNetwork is a network with external router.
+type ExternalNetwork struct {
+	networks.Network
+	external.NetworkExternalExt
+}
+
+// GetFloatingIPNetworkIDByName searches for the external network ID by the provided name.
+func GetFloatingIPNetworkIDByName(client *gophercloud.ServiceClient, networkName string) (string, error) {
+	var externalNetworks []ExternalNetwork
+
+	allPages, err := networks.List(client, networks.ListOpts{
+		Name: networkName,
+	}).AllPages()
+	if err != nil {
+		return "", err
+	}
+
+	if err := networks.ExtractNetworksInto(allPages, &externalNetworks); err != nil {
+		return "", err
+	}
+
+	if len(externalNetworks) == 0 {
+		return "", fmt.Errorf("can't find external network %s", networkName)
+	}
+	// Check and return the first external network.
+	if !externalNetworks[0].External {
+		return "", fmt.Errorf("network %s is not external", networkName)
+	}
+
+	return externalNetworks[0].ID, nil
+}

builder/openstack/run_config.go

@@ -18,23 +18,32 @@ type RunConfig struct {
 	SSHInterface         string              `mapstructure:"ssh_interface"`
 	SSHIPVersion         string              `mapstructure:"ssh_ip_version"`
 
-	SourceImage      string            `mapstructure:"source_image"`
+	SourceImage       string            `mapstructure:"source_image"`
-	SourceImageName  string            `mapstructure:"source_image_name"`
+	SourceImageName   string            `mapstructure:"source_image_name"`
-	Flavor           string            `mapstructure:"flavor"`
+	Flavor            string            `mapstructure:"flavor"`
-	AvailabilityZone string            `mapstructure:"availability_zone"`
+	AvailabilityZone  string            `mapstructure:"availability_zone"`
-	RackconnectWait  bool              `mapstructure:"rackconnect_wait"`
+	RackconnectWait   bool              `mapstructure:"rackconnect_wait"`
-	FloatingIpPool   string            `mapstructure:"floating_ip_pool"`
+	FloatingIPNetwork string            `mapstructure:"floating_ip_network"`
-	FloatingIp       string            `mapstructure:"floating_ip"`
+	FloatingIP        string            `mapstructure:"floating_ip"`
-	ReuseIps         bool              `mapstructure:"reuse_ips"`
+	ReuseIPs          bool              `mapstructure:"reuse_ips"`
-	SecurityGroups   []string          `mapstructure:"security_groups"`
+	SecurityGroups    []string          `mapstructure:"security_groups"`
-	Networks         []string          `mapstructure:"networks"`
+	Networks          []string          `mapstructure:"networks"`
-	UserData         string            `mapstructure:"user_data"`
+	Ports             []string          `mapstructure:"ports"`
-	UserDataFile     string            `mapstructure:"user_data_file"`
+	UserData          string            `mapstructure:"user_data"`
-	InstanceName     string            `mapstructure:"instance_name"`
+	UserDataFile      string            `mapstructure:"user_data_file"`
-	InstanceMetadata map[string]string `mapstructure:"instance_metadata"`
+	InstanceName      string            `mapstructure:"instance_name"`
+	InstanceMetadata  map[string]string `mapstructure:"instance_metadata"`
 
 	ConfigDrive bool `mapstructure:"config_drive"`
 
+	// Used for BC, value will be passed to the "floating_ip_network"
+	FloatingIPPool string `mapstructure:"floating_ip_pool"`
+
+	UseBlockStorageVolume  bool   `mapstructure:"use_blockstorage_volume"`
+	VolumeName             string `mapstructure:"volume_name"`
+	VolumeType             string `mapstructure:"volume_type"`
+	VolumeAvailabilityZone string `mapstructure:"volume_availability_zone"`
+
 	// Not really used, but here for BC
 	OpenstackProvider string `mapstructure:"openstack_provider"`
 	UseFloatingIp     bool   `mapstructure:"use_floating_ip"`
@@ -51,8 +60,8 @@ func (c *RunConfig) Prepare(ctx *interpolate.Context) []error {
 		c.TemporaryKeyPairName = fmt.Sprintf("packer_%s", uuid.TimeOrderedUUID())
 	}
 
-	if c.UseFloatingIp && c.FloatingIpPool == "" {
+	if c.FloatingIPPool != "" && c.FloatingIPNetwork == "" {
-		c.FloatingIpPool = "public"
+		c.FloatingIPNetwork = c.FloatingIPPool
 	}
 
 	// Validation
@@ -89,5 +98,18 @@ func (c *RunConfig) Prepare(ctx *interpolate.Context) []error {
 		}
 	}
 
+	if c.UseBlockStorageVolume {
+		// Use Compute instance availability zone for the Block Storage volume if
+		// it's not provided.
+		if c.VolumeAvailabilityZone == "" {
+			c.VolumeAvailabilityZone = c.AvailabilityZone
+		}
+
+		// Use random name for the Block Storage volume if it's not provided.
+		if c.VolumeName == "" {
+			c.VolumeName = fmt.Sprintf("packer_%s", uuid.TimeOrderedUUID())
+		}
+	}
+
 	return errs
 }

builder/openstack/run_config_test.go

@@ -70,3 +70,60 @@ func TestRunConfigPrepare_SSHPort(t *testing.T) {
 		t.Fatalf("invalid value: %d", c.Comm.SSHPort)
 	}
 }
+
+func TestRunConfigPrepare_BlockStorage(t *testing.T) {
+	c := testRunConfig()
+	c.UseBlockStorageVolume = true
+	c.VolumeType = "fast"
+	if err := c.Prepare(nil); len(err) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+	if c.VolumeType != "fast" {
+		t.Fatalf("invalid value: %s", c.VolumeType)
+	}
+
+	c.AvailabilityZone = "RegionTwo"
+	if err := c.Prepare(nil); len(err) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+
+	if c.VolumeAvailabilityZone != "RegionTwo" {
+		t.Fatalf("invalid value: %s", c.VolumeAvailabilityZone)
+	}
+
+	c.VolumeAvailabilityZone = "RegionOne"
+	if err := c.Prepare(nil); len(err) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+
+	if c.VolumeAvailabilityZone != "RegionOne" {
+		t.Fatalf("invalid value: %s", c.VolumeAvailabilityZone)
+	}
+
+	c.VolumeName = "PackerVolume"
+	if c.VolumeName != "PackerVolume" {
+		t.Fatalf("invalid value: %s", c.VolumeName)
+	}
+}
+
+func TestRunConfigPrepare_FloatingIPPoolCompat(t *testing.T) {
+	c := testRunConfig()
+	c.FloatingIPPool = "uuid1"
+	if err := c.Prepare(nil); len(err) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+
+	if c.FloatingIPNetwork != "uuid1" {
+		t.Fatalf("invalid value: %s", c.FloatingIPNetwork)
+	}
+
+	c.FloatingIPNetwork = "uuid2"
+	c.FloatingIPPool = "uuid3"
+	if err := c.Prepare(nil); len(err) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+
+	if c.FloatingIPNetwork != "uuid2" {
+		t.Fatalf("invalid value: %s", c.FloatingIPNetwork)
+	}
+}

builder/openstack/ssh.go

@@ -9,8 +9,8 @@ import (
 	"time"
 
 	"github.com/gophercloud/gophercloud"
-	"github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/floatingips"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
+	"github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/layer3/floatingips"
 	packerssh "github.com/hashicorp/packer/communicator/ssh"
 	"github.com/hashicorp/packer/helper/multistep"
 	"golang.org/x/crypto/ssh"
@@ -35,9 +35,9 @@ func CommHost(
 
 		// If we have a floating IP, use that
 		ip := state.Get("access_ip").(*floatingips.FloatingIP)
-		if ip != nil && ip.IP != "" {
+		if ip != nil && ip.FloatingIP != "" {
-			log.Printf("[DEBUG] Using floating IP %s to connect", ip.IP)
+			log.Printf("[DEBUG] Using floating IP %s to connect", ip.FloatingIP)
-			return ip.IP, nil
+			return ip.FloatingIP, nil
 		}
 
 		if s.AccessIPv4 != "" {

builder/openstack/step_allocate_ip.go

@@ -4,17 +4,16 @@ import (
 	"context"
 	"fmt"
 
-	"github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/floatingips"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
-	"github.com/gophercloud/gophercloud/pagination"
+	"github.com/gophercloud/gophercloud/openstack/networking/v2/extensions/layer3/floatingips"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 )
 
 type StepAllocateIp struct {
-	FloatingIpPool string
+	FloatingIPNetwork string
-	FloatingIp     string
+	FloatingIP        string
-	ReuseIps       bool
+	ReuseIPs          bool
 }
 
 func (s *StepAllocateIp) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
@@ -23,123 +22,146 @@ func (s *StepAllocateIp) Run(_ context.Context, state multistep.StateBag) multis
 	server := state.Get("server").(*servers.Server)
 
 	// We need the v2 compute client
-	client, err := config.computeV2Client()
+	computeClient, err := config.computeV2Client()
 	if err != nil {
 		err = fmt.Errorf("Error initializing compute client: %s", err)
 		state.Put("error", err)
 		return multistep.ActionHalt
 	}
 
-	var instanceIp floatingips.FloatingIP
+	// We need the v2 network client
+	networkClient, err := config.networkV2Client()
+	if err != nil {
+		err = fmt.Errorf("Error initializing network client: %s", err)
+		state.Put("error", err)
+		return multistep.ActionHalt
+	}
+
+	var instanceIP floatingips.FloatingIP
 
 	// This is here in case we error out before putting instanceIp into the
 	// statebag below, because it is requested by Cleanup()
-	state.Put("access_ip", &instanceIp)
+	state.Put("access_ip", &instanceIP)
 
-	if s.FloatingIp != "" {
+	// Try to Use the OpenStack floating IP by checking provided parameters in
-		instanceIp.IP = s.FloatingIp
+	// the following order:
-	} else if s.FloatingIpPool != "" {
+	//  - try to use "FloatingIP" ID directly if it's provided
-		// If ReuseIps is set to true and we have a free floating IP in
+	//  - try to find free floating IP in the project if "ReuseIPs" is set
-		// the pool, use it first rather than creating one
+	//  - create a new floating IP if "FloatingIPNetwork" is provided (it can be
-		if s.ReuseIps {
+	//    ID or name of the network).
-			ui.Say(fmt.Sprintf("Searching for unassociated floating IP in pool %s", s.FloatingIpPool))
+	if s.FloatingIP != "" {
-			pager := floatingips.List(client)
+		// Try to use FloatingIP if it was provided by the user.
-			err := pager.EachPage(func(page pagination.Page) (bool, error) {
+		freeFloatingIP, err := CheckFloatingIP(networkClient, s.FloatingIP)
-				candidates, err := floatingips.ExtractFloatingIPs(page)
+		if err != nil {
-
+			err := fmt.Errorf("Error using provided floating IP '%s': %s", s.FloatingIP, err)
-				if err != nil {
+			state.Put("error", err)
-					return false, err // stop and throw error out
+			ui.Error(err.Error())
-				}
+			return multistep.ActionHalt
-
-				for _, candidate := range candidates {
-					if candidate.Pool != s.FloatingIpPool || candidate.InstanceID != "" {
-						continue // move to next in list
-					}
-
-					// In correct pool and able to be allocated
-					instanceIp.IP = candidate.IP
-					ui.Message(fmt.Sprintf("Selected floating IP: %s", instanceIp.IP))
-					state.Put("floatingip_istemp", false)
-					return false, nil // stop iterating over pages
-				}
-				return true, nil // try the next page
-			})
-
-			if err != nil {
-				err := fmt.Errorf("Error searching for floating ip from pool '%s'", s.FloatingIpPool)
-				state.Put("error", err)
-				ui.Error(err.Error())
-				return multistep.ActionHalt
-			}
 		}
 
-		if instanceIp.IP == "" {
+		instanceIP = *freeFloatingIP
-			ui.Say(fmt.Sprintf("Creating floating IP..."))
+		ui.Message(fmt.Sprintf("Selected floating IP: '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
-			ui.Message(fmt.Sprintf("Pool: %s", s.FloatingIpPool))
+		state.Put("floatingip_istemp", false)
-			newIp, err := floatingips.Create(client, floatingips.CreateOpts{
+	} else if s.ReuseIPs {
-				Pool: s.FloatingIpPool,
+		// If ReuseIPs is set to true and we have a free floating IP, use it rather
-			}).Extract()
+		// than creating one.
-			if err != nil {
+		ui.Say(fmt.Sprint("Searching for unassociated floating IP"))
-				err := fmt.Errorf("Error creating floating ip from pool '%s'", s.FloatingIpPool)
+		freeFloatingIP, err := FindFreeFloatingIP(networkClient)
-				state.Put("error", err)
+		if err != nil {
-				ui.Error(err.Error())
+			err := fmt.Errorf("Error searching for floating IP: %s", err)
-				return multistep.ActionHalt
+			state.Put("error", err)
-			}
+			ui.Error(err.Error())
-
+			return multistep.ActionHalt
-			instanceIp = *newIp
-			ui.Message(fmt.Sprintf("Created floating IP: %s", instanceIp.IP))
-			state.Put("floatingip_istemp", true)
 		}
+
+		instanceIP = *freeFloatingIP
+		ui.Message(fmt.Sprintf("Selected floating IP: '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
+		state.Put("floatingip_istemp", false)
+	} else if s.FloatingIPNetwork != "" {
+		// Lastly, if FloatingIPNetwork was provided by the user, we need to use it
+		// to allocate a new floating IP and associate it to the instance.
+		floatingNetwork, err := CheckFloatingIPNetwork(networkClient, s.FloatingIPNetwork)
+		if err != nil {
+			err := fmt.Errorf("Error using the provided floating_ip_network: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
+
+		ui.Say(fmt.Sprintf("Creating floating IP using network %s ...", floatingNetwork))
+		newIP, err := floatingips.Create(networkClient, floatingips.CreateOpts{
+			FloatingNetworkID: floatingNetwork,
+		}).Extract()
+		if err != nil {
+			err := fmt.Errorf("Error creating floating IP from floating network '%s': %s", floatingNetwork, err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
+
+		instanceIP = *newIP
+		ui.Message(fmt.Sprintf("Created floating IP: '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
+		state.Put("floatingip_istemp", true)
 	}
 
-	if instanceIp.IP != "" {
+	// Assoctate a floating IP if it was obtained in the previous steps.
-		ui.Say(fmt.Sprintf("Associating floating IP with server..."))
+	if instanceIP.ID != "" {
-		ui.Message(fmt.Sprintf("IP: %s", instanceIp.IP))
+		ui.Say(fmt.Sprintf("Associating floating IP '%s' (%s) with instance port...",
-		err := floatingips.AssociateInstance(client, server.ID, floatingips.AssociateOpts{
+			instanceIP.ID, instanceIP.FloatingIP))
-			FloatingIP: instanceIp.IP,
+
-		}).ExtractErr()
+		portID, err := GetInstancePortID(computeClient, server.ID)
+		if err != nil {
+			err := fmt.Errorf("Error getting interfaces of the instance '%s': %s", server.ID, err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
+
+		_, err = floatingips.Update(networkClient, instanceIP.ID, floatingips.UpdateOpts{
+			PortID: &portID,
+		}).Extract()
 		if err != nil {
 			err := fmt.Errorf(
-				"Error associating floating IP %s with instance: %s",
+				"Error associating floating IP '%s' (%s) with instance port '%s': %s",
-				instanceIp.IP, err)
+				instanceIP.ID, instanceIP.FloatingIP, portID, err)
 			state.Put("error", err)
 			ui.Error(err.Error())
 			return multistep.ActionHalt
 		}
 
 		ui.Message(fmt.Sprintf(
-			"Added floating IP %s to instance!", instanceIp.IP))
+			"Added floating IP '%s' (%s) to instance!", instanceIP.ID, instanceIP.FloatingIP))
 	}
 
-	state.Put("access_ip", &instanceIp)
+	state.Put("access_ip", &instanceIP)
 	return multistep.ActionContinue
 }
 
 func (s *StepAllocateIp) Cleanup(state multistep.StateBag) {
 	config := state.Get("config").(Config)
 	ui := state.Get("ui").(packer.Ui)
-	instanceIp := state.Get("access_ip").(*floatingips.FloatingIP)
+	instanceIP := state.Get("access_ip").(*floatingips.FloatingIP)
 
 	// Don't delete pool addresses we didn't allocate
 	if state.Get("floatingip_istemp") == false {
 		return
 	}
 
-	// We need the v2 compute client
+	// We need the v2 network client
-	client, err := config.computeV2Client()
+	client, err := config.networkV2Client()
 	if err != nil {
 		ui.Error(fmt.Sprintf(
-			"Error deleting temporary floating IP %s", instanceIp.IP))
+			"Error deleting temporary floating IP '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
 		return
 	}
 
-	if s.FloatingIpPool != "" && instanceIp.ID != "" {
+	if instanceIP.ID != "" {
-		if err := floatingips.Delete(client, instanceIp.ID).ExtractErr(); err != nil {
+		if err := floatingips.Delete(client, instanceIP.ID).ExtractErr(); err != nil {
 			ui.Error(fmt.Sprintf(
-				"Error deleting temporary floating IP %s", instanceIp.IP))
+				"Error deleting temporary floating IP '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
 			return
 		}
 
-		ui.Say(fmt.Sprintf("Deleted temporary floating IP %s", instanceIp.IP))
+		ui.Say(fmt.Sprintf("Deleted temporary floating IP '%s' (%s)", instanceIP.ID, instanceIP.FloatingIP))
 	}
 }

builder/openstack/step_create_image.go

@@ -6,6 +6,8 @@ import (
 	"log"
 	"time"
 
+	"github.com/gophercloud/gophercloud/openstack/blockstorage/extensions/volumeactions"
+
 	"github.com/gophercloud/gophercloud"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/images"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
@@ -13,7 +15,9 @@ import (
 	"github.com/hashicorp/packer/packer"
 )
 
-type stepCreateImage struct{}
+type stepCreateImage struct {
+	UseBlockStorageVolume bool
+}
 
 func (s *stepCreateImage) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
 	config := state.Get("config").(Config)
@@ -28,17 +32,41 @@ func (s *stepCreateImage) Run(_ context.Context, state multistep.StateBag) multi
 		return multistep.ActionHalt
 	}
 
-	// Create the image
+	// Create the image.
+	// Image source depends on the type of the Compute instance. It can be
+	// Block Storage service volume or regular Compute service local volume.
 	ui.Say(fmt.Sprintf("Creating the image: %s", config.ImageName))
-	imageId, err := servers.CreateImage(client, server.ID, servers.CreateImageOpts{
+	var imageId string
-		Name:     config.ImageName,
+	if s.UseBlockStorageVolume {
-		Metadata: config.ImageMetadata,
+		// We need the v3 block storage client.
-	}).ExtractImageID()
+		blockStorageClient, err := config.blockStorageV3Client()
-	if err != nil {
+		if err != nil {
-		err := fmt.Errorf("Error creating image: %s", err)
+			err = fmt.Errorf("Error initializing block storage client: %s", err)
-		state.Put("error", err)
+			state.Put("error", err)
-		ui.Error(err.Error())
+			return multistep.ActionHalt
-		return multistep.ActionHalt
+		}
+		volume := state.Get("volume_id").(string)
+		image, err := volumeactions.UploadImage(blockStorageClient, volume, volumeactions.UploadImageOpts{
+			ImageName: config.ImageName,
+		}).Extract()
+		if err != nil {
+			err := fmt.Errorf("Error creating image: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
+		imageId = image.ImageID
+	} else {
+		imageId, err = servers.CreateImage(client, server.ID, servers.CreateImageOpts{
+			Name:     config.ImageName,
+			Metadata: config.ImageMetadata,
+		}).ExtractImageID()
+		if err != nil {
+			err := fmt.Errorf("Error creating image: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
 	}
 
 	// Set the Image ID in the state

builder/openstack/step_create_volume.go

@@ -0,0 +1,111 @@
+package openstack
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/gophercloud/gophercloud/openstack/blockstorage/v3/volumes"
+	"github.com/hashicorp/packer/helper/multistep"
+	"github.com/hashicorp/packer/packer"
+)
+
+type StepCreateVolume struct {
+	UseBlockStorageVolume  bool
+	SourceImage            string
+	VolumeName             string
+	VolumeType             string
+	VolumeAvailabilityZone string
+	volumeID               string
+	doCleanup              bool
+}
+
+func (s *StepCreateVolume) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+	// Proceed only if block storage volume is required.
+	if !s.UseBlockStorageVolume {
+		return multistep.ActionContinue
+	}
+
+	config := state.Get("config").(Config)
+	ui := state.Get("ui").(packer.Ui)
+
+	// We will need Block Storage and Image services clients.
+	blockStorageClient, err := config.blockStorageV3Client()
+	if err != nil {
+		err = fmt.Errorf("Error initializing block storage client: %s", err)
+		state.Put("error", err)
+		return multistep.ActionHalt
+	}
+	imageClient, err := config.imageV2Client()
+	if err != nil {
+		err = fmt.Errorf("Error initializing image client: %s", err)
+		state.Put("error", err)
+		return multistep.ActionHalt
+	}
+
+	// Get needed volume size from the source image.
+	volumeSize, err := GetVolumeSize(imageClient, s.SourceImage)
+	if err != nil {
+		err := fmt.Errorf("Error creating volume: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+
+	ui.Say("Creating volume...")
+	volumeOpts := volumes.CreateOpts{
+		Size:             volumeSize,
+		VolumeType:       s.VolumeType,
+		AvailabilityZone: s.VolumeAvailabilityZone,
+		Name:             s.VolumeName,
+		ImageID:          s.SourceImage,
+	}
+	volume, err := volumes.Create(blockStorageClient, volumeOpts).Extract()
+	if err != nil {
+		err := fmt.Errorf("Error creating volume: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+
+	// Wait for volume to become available.
+	ui.Say(fmt.Sprintf("Waiting for volume %s (volume id: %s) to become available...", config.VolumeName, volume.ID))
+	if err := WaitForVolume(blockStorageClient, volume.ID); err != nil {
+		err := fmt.Errorf("Error waiting for volume: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+
+	// Volume was created, so remember to clean it up.
+	s.doCleanup = true
+
+	// Set the Volume ID in the state.
+	ui.Message(fmt.Sprintf("Volume ID: %s", volume.ID))
+	state.Put("volume_id", volume.ID)
+	s.volumeID = volume.ID
+
+	return multistep.ActionContinue
+}
+
+func (s *StepCreateVolume) Cleanup(state multistep.StateBag) {
+	if !s.doCleanup {
+		return
+	}
+
+	config := state.Get("config").(Config)
+	ui := state.Get("ui").(packer.Ui)
+
+	blockStorageClient, err := config.blockStorageV3Client()
+	if err != nil {
+		ui.Error(fmt.Sprintf(
+			"Error cleaning up volume. Please delete the volume manually: %s", s.volumeID))
+		return
+	}
+
+	ui.Say(fmt.Sprintf("Deleting volume: %s ...", s.volumeID))
+	err = volumes.Delete(blockStorageClient, s.volumeID).ExtractErr()
+	if err != nil {
+		ui.Error(fmt.Sprintf(
+			"Error cleaning up volume. Please delete the volume manually: %s", s.volumeID))
+	}
+}

builder/openstack/step_detach_volume.go

@@ -0,0 +1,54 @@
+package openstack
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/gophercloud/gophercloud/openstack/blockstorage/extensions/volumeactions"
+	"github.com/hashicorp/packer/helper/multistep"
+	"github.com/hashicorp/packer/packer"
+)
+
+type StepDetachVolume struct {
+	UseBlockStorageVolume bool
+}
+
+func (s *StepDetachVolume) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+	// Proceed only if block storage volume is used.
+	if !s.UseBlockStorageVolume {
+		return multistep.ActionContinue
+	}
+
+	config := state.Get("config").(Config)
+	ui := state.Get("ui").(packer.Ui)
+
+	blockStorageClient, err := config.blockStorageV3Client()
+	if err != nil {
+		err = fmt.Errorf("Error initializing block storage client: %s", err)
+		state.Put("error", err)
+		return multistep.ActionHalt
+	}
+
+	volume := state.Get("volume_id").(string)
+	ui.Say(fmt.Sprintf("Detaching volume %s (volume id: %s)", config.VolumeName, volume))
+	if err := volumeactions.Detach(blockStorageClient, volume, volumeactions.DetachOpts{}).ExtractErr(); err != nil {
+		err = fmt.Errorf("Error detaching block storage volume: %s", err)
+		state.Put("error", err)
+		return multistep.ActionHalt
+	}
+
+	// Wait for volume to become available.
+	ui.Say(fmt.Sprintf("Waiting for volume %s (volume id: %s) to become available...", config.VolumeName, volume))
+	if err := WaitForVolume(blockStorageClient, volume); err != nil {
+		err := fmt.Errorf("Error waiting for volume: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+
+	return multistep.ActionContinue
+}
+
+func (s *StepDetachVolume) Cleanup(multistep.StateBag) {
+	// No cleanup.
+}

builder/openstack/step_run_source_server.go

@@ -6,6 +6,7 @@ import (
 	"io/ioutil"
 	"log"
 
+	"github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/bootfromvolume"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/extensions/keypairs"
 	"github.com/gophercloud/gophercloud/openstack/compute/v2/servers"
 	"github.com/hashicorp/packer/helper/multistep"
@@ -13,17 +14,19 @@ import (
 )
 
 type StepRunSourceServer struct {
-	Name             string
+	Name                  string
-	SourceImage      string
+	SourceImage           string
-	SourceImageName  string
+	SourceImageName       string
-	SecurityGroups   []string
+	SecurityGroups        []string
-	Networks         []string
+	Networks              []string
-	AvailabilityZone string
+	Ports                 []string
-	UserData         string
+	AvailabilityZone      string
-	UserDataFile     string
+	UserData              string
-	ConfigDrive      bool
+	UserDataFile          string
-	InstanceMetadata map[string]string
+	ConfigDrive           bool
-	server           *servers.Server
+	InstanceMetadata      map[string]string
+	UseBlockStorageVolume bool
+	server                *servers.Server
 }
 
 func (s *StepRunSourceServer) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
@@ -39,9 +42,13 @@ func (s *StepRunSourceServer) Run(_ context.Context, state multistep.StateBag) m
 		return multistep.ActionHalt
 	}
 
-	networks := make([]servers.Network, len(s.Networks))
+	networks := make([]servers.Network, len(s.Networks)+len(s.Ports))
-	for i, networkUuid := range s.Networks {
+	i := 0
-		networks[i].UUID = networkUuid
+	for ; i < len(s.Ports); i++ {
+		networks[i].Port = s.Ports[i]
+	}
+	for ; i < len(networks); i++ {
+		networks[i].UUID = s.Networks[i]
 	}
 
 	userData := []byte(s.UserData)
@@ -69,18 +76,40 @@ func (s *StepRunSourceServer) Run(_ context.Context, state multistep.StateBag) m
 		ServiceClient:    computeClient,
 		Metadata:         s.InstanceMetadata,
 	}
-
 	var serverOptsExt servers.CreateOptsBuilder
-	keyName, hasKey := state.GetOk("keyPair")
+
-	if hasKey {
+	// Create root volume in the Block Storage service if required.
-		serverOptsExt = keypairs.CreateOptsExt{
+	// Add block device mapping v2 to the server create options if required.
+	if s.UseBlockStorageVolume {
+		volume := state.Get("volume_id").(string)
+		blockDeviceMappingV2 := []bootfromvolume.BlockDevice{
+			{
+				BootIndex:       0,
+				DestinationType: bootfromvolume.DestinationVolume,
+				SourceType:      bootfromvolume.SourceVolume,
+				UUID:            volume,
+			},
+		}
+		// ImageRef and block device mapping is an invalid options combination.
+		serverOpts.ImageRef = ""
+		serverOptsExt = bootfromvolume.CreateOptsExt{
 			CreateOptsBuilder: serverOpts,
-			KeyName:           keyName.(string),
+			BlockDevice:       blockDeviceMappingV2,
 		}
 	} else {
 		serverOptsExt = serverOpts
 	}
 
+	// Add keypair to the server create options.
+	keyName, hasKey := state.GetOk("keyPair")
+	if hasKey {
+		serverOptsExt = keypairs.CreateOptsExt{
+			CreateOptsBuilder: serverOptsExt,
+			KeyName:           keyName.(string),
+		}
+	}
+
+	ui.Say("Launching server...")
 	s.server, err = servers.Create(computeClient, serverOptsExt).Extract()
 	if err != nil {
 		err := fmt.Errorf("Error launching source server: %s", err)

builder/openstack/volume.go

@@ -0,0 +1,67 @@
+package openstack
+
+import (
+	"log"
+	"time"
+
+	"github.com/gophercloud/gophercloud"
+	"github.com/gophercloud/gophercloud/openstack/blockstorage/v3/volumes"
+	"github.com/gophercloud/gophercloud/openstack/imageservice/v2/images"
+)
+
+// WaitForVolume waits for the given volume to become available.
+func WaitForVolume(blockStorageClient *gophercloud.ServiceClient, volumeID string) error {
+	maxNumErrors := 10
+	numErrors := 0
+
+	for {
+		volume, err := volumes.Get(blockStorageClient, volumeID).Extract()
+		if err != nil {
+			errCode, ok := err.(*gophercloud.ErrUnexpectedResponseCode)
+			if ok && (errCode.Actual == 500 || errCode.Actual == 404) {
+				numErrors++
+				if numErrors >= maxNumErrors {
+					log.Printf("[ERROR] Maximum number of errors (%d) reached; failing with: %s", numErrors, err)
+					return err
+				}
+				log.Printf("[ERROR] %d error received, will ignore and retry: %s", errCode.Actual, err)
+				time.Sleep(2 * time.Second)
+				continue
+			}
+
+			return err
+		}
+
+		if volume.Status == "available" {
+			return nil
+		}
+
+		log.Printf("Waiting for volume creation status: %s", volume.Status)
+		time.Sleep(2 * time.Second)
+	}
+}
+
+// GetVolumeSize returns volume size in gigabytes based on the image min disk
+// value if it's not empty.
+// Or it calculates needed gigabytes size from the image bytes size.
+func GetVolumeSize(imageClient *gophercloud.ServiceClient, imageID string) (int, error) {
+	sourceImage, err := images.Get(imageClient, imageID).Extract()
+	if err != nil {
+		return 0, err
+	}
+
+	if sourceImage.MinDiskGigabytes != 0 {
+		return sourceImage.MinDiskGigabytes, nil
+	}
+
+	volumeSizeMB := sourceImage.SizeBytes / 1024 / 1024
+	volumeSizeGB := int(sourceImage.SizeBytes / 1024 / 1024 / 1024)
+
+	// Increment gigabytes size if the initial size can't be divided without
+	// remainder.
+	if volumeSizeMB%1024 > 0 {
+		volumeSizeGB++
+	}
+
+	return volumeSizeGB, nil
+}

builder/oracle/oci/artifact.go

@@ -1,6 +1,7 @@
 package oci
 
 import (
+	"context"
 	"fmt"
 
 	"github.com/oracle/oci-go-sdk/core"
@@ -41,11 +42,12 @@ func (a *Artifact) String() string {
 	)
 }
 
+// State ...
 func (a *Artifact) State(name string) interface{} {
 	return nil
 }
 
 // Destroy deletes the custom image associated with the artifact.
 func (a *Artifact) Destroy() error {
-	return a.driver.DeleteImage(*a.Image.Id)
+	return a.driver.DeleteImage(context.TODO(), *a.Image.Id)
 }

builder/oracle/oci/config.go

@@ -44,6 +44,17 @@ type Config struct {
 	BaseImageID string `mapstructure:"base_image_ocid"`
 	Shape       string `mapstructure:"shape"`
 	ImageName   string `mapstructure:"image_name"`
+
+	// Instance
+	InstanceName string `mapstructure:"instance_name"`
+
+	// Metadata optionally contains custom metadata key/value pairs provided in the
+	// configuration. While this can be used to set metadata["user_data"] the explicit
+	// "user_data" and "user_data_file" values will have precedence.
+	// An instance's metadata can be obtained from at http://169.254.169.254 on the
+	// launched instance.
+	Metadata map[string]string `mapstructure:"metadata"`
+
 	// UserData and UserDataFile file are both optional and mutually exclusive.
 	UserData     string `mapstructure:"user_data"`
 	UserDataFile string `mapstructure:"user_data_file"`

builder/oracle/oci/config_test.go

@@ -29,11 +29,14 @@ func testConfig(accessConfFile *os.File) map[string]interface{} {
 		// Comm
 		"ssh_username":   "opc",
 		"use_private_ip": false,
+		"metadata": map[string]string{
+			"key": "value",
+		},
 	}
 }
 
 func TestConfig(t *testing.T) {
-	// Shared set-up and defered deletion
+	// Shared set-up and deferred deletion
 
 	cfg, keyFile, err := baseTestConfigWithTmpKeyFile()
 	if err != nil {

builder/oracle/oci/driver.go

@@ -1,14 +1,18 @@
 package oci
 
-import "github.com/oracle/oci-go-sdk/core"
+import (
+	"context"
+
+	"github.com/oracle/oci-go-sdk/core"
+)
 
 // Driver interfaces between the builder steps and the OCI SDK.
 type Driver interface {
-	CreateInstance(publicKey string) (string, error)
+	CreateInstance(ctx context.Context, publicKey string) (string, error)
-	CreateImage(id string) (core.Image, error)
+	CreateImage(ctx context.Context, id string) (core.Image, error)
-	DeleteImage(id string) error
+	DeleteImage(ctx context.Context, id string) error
-	GetInstanceIP(id string) (string, error)
+	GetInstanceIP(ctx context.Context, id string) (string, error)
-	TerminateInstance(id string) error
+	TerminateInstance(ctx context.Context, id string) error
-	WaitForImageCreation(id string) error
+	WaitForImageCreation(ctx context.Context, id string) error
-	WaitForInstanceState(id string, waitStates []string, terminalState string) error
+	WaitForInstanceState(ctx context.Context, id string, waitStates []string, terminalState string) error
 }

builder/oracle/oci/driver_mock.go

@@ -1,6 +1,10 @@
 package oci
 
-import "github.com/oracle/oci-go-sdk/core"
+import (
+	"context"
+
+	"github.com/oracle/oci-go-sdk/core"
+)
 
 // driverMock implements the Driver interface and communicates with Oracle
 // OCI.
@@ -27,7 +31,7 @@ type driverMock struct {
 }
 
 // CreateInstance creates a new compute instance.
-func (d *driverMock) CreateInstance(publicKey string) (string, error) {
+func (d *driverMock) CreateInstance(ctx context.Context, publicKey string) (string, error) {
 	if d.CreateInstanceErr != nil {
 		return "", d.CreateInstanceErr
 	}
@@ -38,7 +42,7 @@ func (d *driverMock) CreateInstance(publicKey string) (string, error) {
 }
 
 // CreateImage creates a new custom image.
-func (d *driverMock) CreateImage(id string) (core.Image, error) {
+func (d *driverMock) CreateImage(ctx context.Context, id string) (core.Image, error) {
 	if d.CreateImageErr != nil {
 		return core.Image{}, d.CreateImageErr
 	}
@@ -47,7 +51,7 @@ func (d *driverMock) CreateImage(id string) (core.Image, error) {
 }
 
 // DeleteImage mocks deleting a custom image.
-func (d *driverMock) DeleteImage(id string) error {
+func (d *driverMock) DeleteImage(ctx context.Context, id string) error {
 	if d.DeleteImageErr != nil {
 		return d.DeleteImageErr
 	}
@@ -58,7 +62,7 @@ func (d *driverMock) DeleteImage(id string) error {
 }
 
 // GetInstanceIP returns the public or private IP corresponding to the given instance id.
-func (d *driverMock) GetInstanceIP(id string) (string, error) {
+func (d *driverMock) GetInstanceIP(ctx context.Context, id string) (string, error) {
 	if d.GetInstanceIPErr != nil {
 		return "", d.GetInstanceIPErr
 	}
@@ -69,7 +73,7 @@ func (d *driverMock) GetInstanceIP(id string) (string, error) {
 }
 
 // TerminateInstance terminates a compute instance.
-func (d *driverMock) TerminateInstance(id string) error {
+func (d *driverMock) TerminateInstance(ctx context.Context, id string) error {
 	if d.TerminateInstanceErr != nil {
 		return d.TerminateInstanceErr
 	}
@@ -81,12 +85,12 @@ func (d *driverMock) TerminateInstance(id string) error {
 
 // WaitForImageCreation waits for a provisioning custom image to reach the
 // "AVAILABLE" state.
-func (d *driverMock) WaitForImageCreation(id string) error {
+func (d *driverMock) WaitForImageCreation(ctx context.Context, id string) error {
 	return d.WaitForImageCreationErr
 }
 
 // WaitForInstanceState waits for an instance to reach the a given terminal
 // state.
-func (d *driverMock) WaitForInstanceState(id string, waitStates []string, terminalState string) error {
+func (d *driverMock) WaitForInstanceState(ctx context.Context, id string, waitStates []string, terminalState string) error {
 	return d.WaitForInstanceStateErr
 }

builder/oracle/oci/driver_oci.go

@@ -15,6 +15,7 @@ type driverOCI struct {
 	computeClient core.ComputeClient
 	vcnClient     core.VirtualNetworkClient
 	cfg           *Config
+	context       context.Context
 }
 
 // NewDriverOCI Creates a new driverOCI with a connected compute client and a connected vcn client.
@@ -37,22 +38,34 @@ func NewDriverOCI(cfg *Config) (Driver, error) {
 }
 
 // CreateInstance creates a new compute instance.
-func (d *driverOCI) CreateInstance(publicKey string) (string, error) {
+func (d *driverOCI) CreateInstance(ctx context.Context, publicKey string) (string, error) {
 	metadata := map[string]string{
 		"ssh_authorized_keys": publicKey,
 	}
+	if d.cfg.Metadata != nil {
+		for key, value := range d.cfg.Metadata {
+			metadata[key] = value
+		}
+	}
 	if d.cfg.UserData != "" {
 		metadata["user_data"] = d.cfg.UserData
 	}
 
-	instance, err := d.computeClient.LaunchInstance(context.TODO(), core.LaunchInstanceRequest{LaunchInstanceDetails: core.LaunchInstanceDetails{
+	instanceDetails := core.LaunchInstanceDetails{
 		AvailabilityDomain: &d.cfg.AvailabilityDomain,
 		CompartmentId:      &d.cfg.CompartmentID,
 		ImageId:            &d.cfg.BaseImageID,
 		Shape:              &d.cfg.Shape,
 		SubnetId:           &d.cfg.SubnetID,
 		Metadata:           metadata,
-	}})
+	}
+
+	// When empty, the default display name is used.
+	if d.cfg.InstanceName != "" {
+		instanceDetails.DisplayName = &d.cfg.InstanceName
+	}
+
+	instance, err := d.computeClient.LaunchInstance(context.TODO(), core.LaunchInstanceRequest{LaunchInstanceDetails: instanceDetails})
 
 	if err != nil {
 		return "", err
@@ -62,8 +75,8 @@ func (d *driverOCI) CreateInstance(publicKey string) (string, error) {
 }
 
 // CreateImage creates a new custom image.
-func (d *driverOCI) CreateImage(id string) (core.Image, error) {
+func (d *driverOCI) CreateImage(ctx context.Context, id string) (core.Image, error) {
-	res, err := d.computeClient.CreateImage(context.TODO(), core.CreateImageRequest{CreateImageDetails: core.CreateImageDetails{
+	res, err := d.computeClient.CreateImage(ctx, core.CreateImageRequest{CreateImageDetails: core.CreateImageDetails{
 		CompartmentId: &d.cfg.CompartmentID,
 		InstanceId:    &id,
 		DisplayName:   &d.cfg.ImageName,
@@ -77,14 +90,14 @@ func (d *driverOCI) CreateImage(id string) (core.Image, error) {
 }
 
 // DeleteImage deletes a custom image.
-func (d *driverOCI) DeleteImage(id string) error {
+func (d *driverOCI) DeleteImage(ctx context.Context, id string) error {
-	_, err := d.computeClient.DeleteImage(context.TODO(), core.DeleteImageRequest{ImageId: &id})
+	_, err := d.computeClient.DeleteImage(ctx, core.DeleteImageRequest{ImageId: &id})
 	return err
 }
 
 // GetInstanceIP returns the public or private IP corresponding to the given instance id.
-func (d *driverOCI) GetInstanceIP(id string) (string, error) {
+func (d *driverOCI) GetInstanceIP(ctx context.Context, id string) (string, error) {
-	vnics, err := d.computeClient.ListVnicAttachments(context.TODO(), core.ListVnicAttachmentsRequest{
+	vnics, err := d.computeClient.ListVnicAttachments(ctx, core.ListVnicAttachmentsRequest{
 		InstanceId:    &id,
 		CompartmentId: &d.cfg.CompartmentID,
 	})
@@ -96,7 +109,7 @@ func (d *driverOCI) GetInstanceIP(id string) (string, error) {
 		return "", errors.New("instance has zero VNICs")
 	}
 
-	vnic, err := d.vcnClient.GetVnic(context.TODO(), core.GetVnicRequest{VnicId: vnics.Items[0].VnicId})
+	vnic, err := d.vcnClient.GetVnic(ctx, core.GetVnicRequest{VnicId: vnics.Items[0].VnicId})
 	if err != nil {
 		return "", fmt.Errorf("Error getting VNIC details: %s", err)
 	}
@@ -112,8 +125,8 @@ func (d *driverOCI) GetInstanceIP(id string) (string, error) {
 	return *vnic.PublicIp, nil
 }
 
-func (d *driverOCI) GetInstanceInitialCredentials(id string) (string, string, error) {
+func (d *driverOCI) GetInstanceInitialCredentials(ctx context.Context, id string) (string, string, error) {
-	credentials, err := d.computeClient.GetWindowsInstanceInitialCredentials(context.TODO(), core.GetWindowsInstanceInitialCredentialsRequest{
+	credentials, err := d.computeClient.GetWindowsInstanceInitialCredentials(ctx, core.GetWindowsInstanceInitialCredentialsRequest{
 		InstanceId: &id,
 	})
 	if err != nil {
@@ -124,8 +137,8 @@ func (d *driverOCI) GetInstanceInitialCredentials(id string) (string, string, er
 }
 
 // TerminateInstance terminates a compute instance.
-func (d *driverOCI) TerminateInstance(id string) error {
+func (d *driverOCI) TerminateInstance(ctx context.Context, id string) error {
-	_, err := d.computeClient.TerminateInstance(context.TODO(), core.TerminateInstanceRequest{
+	_, err := d.computeClient.TerminateInstance(ctx, core.TerminateInstanceRequest{
 		InstanceId: &id,
 	})
 	return err
@@ -133,10 +146,10 @@ func (d *driverOCI) TerminateInstance(id string) error {
 
 // WaitForImageCreation waits for a provisioning custom image to reach the
 // "AVAILABLE" state.
-func (d *driverOCI) WaitForImageCreation(id string) error {
+func (d *driverOCI) WaitForImageCreation(ctx context.Context, id string) error {
 	return waitForResourceToReachState(
 		func(string) (string, error) {
-			image, err := d.computeClient.GetImage(context.TODO(), core.GetImageRequest{ImageId: &id})
+			image, err := d.computeClient.GetImage(ctx, core.GetImageRequest{ImageId: &id})
 			if err != nil {
 				return "", err
 			}
@@ -152,10 +165,10 @@ func (d *driverOCI) WaitForImageCreation(id string) error {
 
 // WaitForInstanceState waits for an instance to reach the a given terminal
 // state.
-func (d *driverOCI) WaitForInstanceState(id string, waitStates []string, terminalState string) error {
+func (d *driverOCI) WaitForInstanceState(ctx context.Context, id string, waitStates []string, terminalState string) error {
 	return waitForResourceToReachState(
 		func(string) (string, error) {
-			instance, err := d.computeClient.GetInstance(context.TODO(), core.GetInstanceRequest{InstanceId: &id})
+			instance, err := d.computeClient.GetInstance(ctx, core.GetInstanceRequest{InstanceId: &id})
 			if err != nil {
 				return "", err
 			}

builder/oracle/oci/step_create_instance.go

@@ -10,7 +10,7 @@ import (
 
 type stepCreateInstance struct{}
 
-func (s *stepCreateInstance) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *stepCreateInstance) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	var (
 		driver    = state.Get("driver").(Driver)
 		ui        = state.Get("ui").(packer.Ui)
@@ -19,7 +19,7 @@ func (s *stepCreateInstance) Run(_ context.Context, state multistep.StateBag) mu
 
 	ui.Say("Creating instance...")
 
-	instanceID, err := driver.CreateInstance(publicKey)
+	instanceID, err := driver.CreateInstance(ctx, publicKey)
 	if err != nil {
 		err = fmt.Errorf("Problem creating instance: %s", err)
 		ui.Error(err.Error())
@@ -33,7 +33,7 @@ func (s *stepCreateInstance) Run(_ context.Context, state multistep.StateBag) mu
 
 	ui.Say("Waiting for instance to enter 'RUNNING' state...")
 
-	if err = driver.WaitForInstanceState(instanceID, []string{"STARTING", "PROVISIONING"}, "RUNNING"); err != nil {
+	if err = driver.WaitForInstanceState(ctx, instanceID, []string{"STARTING", "PROVISIONING"}, "RUNNING"); err != nil {
 		err = fmt.Errorf("Error waiting for instance to start: %s", err)
 		ui.Error(err.Error())
 		state.Put("error", err)
@@ -57,14 +57,14 @@ func (s *stepCreateInstance) Cleanup(state multistep.StateBag) {
 
 	ui.Say(fmt.Sprintf("Terminating instance (%s)...", id))
 
-	if err := driver.TerminateInstance(id); err != nil {
+	if err := driver.TerminateInstance(context.TODO(), id); err != nil {
 		err = fmt.Errorf("Error terminating instance. Please terminate manually: %s", err)
 		ui.Error(err.Error())
 		state.Put("error", err)
 		return
 	}
 
-	err := driver.WaitForInstanceState(id, []string{"TERMINATING"}, "TERMINATED")
+	err := driver.WaitForInstanceState(context.TODO(), id, []string{"TERMINATING"}, "TERMINATED")
 	if err != nil {
 		err = fmt.Errorf("Error terminating instance. Please terminate manually: %s", err)
 		ui.Error(err.Error())

builder/oracle/oci/step_get_default_credentials.go

@@ -17,7 +17,7 @@ type stepGetDefaultCredentials struct {
 	BuildName string
 }
 
-func (s *stepGetDefaultCredentials) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *stepGetDefaultCredentials) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	var (
 		driver = state.Get("driver").(*driverOCI)
 		ui     = state.Get("ui").(packer.Ui)
@@ -36,7 +36,7 @@ func (s *stepGetDefaultCredentials) Run(_ context.Context, state multistep.State
 		return multistep.ActionContinue
 	}
 
-	username, password, err := driver.GetInstanceInitialCredentials(id)
+	username, password, err := driver.GetInstanceInitialCredentials(ctx, id)
 	if err != nil {
 		err = fmt.Errorf("Error getting instance's credentials: %s", err)
 		ui.Error(err.Error())

builder/oracle/oci/step_image.go

@@ -10,7 +10,7 @@ import (
 
 type stepImage struct{}
 
-func (s *stepImage) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *stepImage) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	var (
 		driver     = state.Get("driver").(Driver)
 		ui         = state.Get("ui").(packer.Ui)
@@ -19,7 +19,7 @@ func (s *stepImage) Run(_ context.Context, state multistep.StateBag) multistep.S
 
 	ui.Say("Creating image from instance...")
 
-	image, err := driver.CreateImage(instanceID)
+	image, err := driver.CreateImage(ctx, instanceID)
 	if err != nil {
 		err = fmt.Errorf("Error creating image from instance: %s", err)
 		ui.Error(err.Error())
@@ -27,7 +27,7 @@ func (s *stepImage) Run(_ context.Context, state multistep.StateBag) multistep.S
 		return multistep.ActionHalt
 	}
 
-	err = driver.WaitForImageCreation(*image.Id)
+	err = driver.WaitForImageCreation(ctx, *image.Id)
 	if err != nil {
 		err = fmt.Errorf("Error waiting for image creation to finish: %s", err)
 		ui.Error(err.Error())

builder/oracle/oci/step_instance_info.go

@@ -10,14 +10,14 @@ import (
 
 type stepInstanceInfo struct{}
 
-func (s *stepInstanceInfo) Run(_ context.Context, state multistep.StateBag) multistep.StepAction {
+func (s *stepInstanceInfo) Run(ctx context.Context, state multistep.StateBag) multistep.StepAction {
 	var (
 		driver = state.Get("driver").(Driver)
 		ui     = state.Get("ui").(packer.Ui)
 		id     = state.Get("instance_id").(string)
 	)
 
-	ip, err := driver.GetInstanceIP(id)
+	ip, err := driver.GetInstanceIP(ctx, id)
 	if err != nil {
 		err = fmt.Errorf("Error getting instance's IP: %s", err)
 		ui.Error(err.Error())

builder/qemu/builder.go

@@ -99,6 +99,7 @@ type Config struct {
 	Format            string     `mapstructure:"format"`
 	Headless          bool       `mapstructure:"headless"`
 	DiskImage         bool       `mapstructure:"disk_image"`
+	UseBackingFile    bool       `mapstructure:"use_backing_file"`
 	MachineType       string     `mapstructure:"machine_type"`
 	NetDevice         string     `mapstructure:"net_device"`
 	OutputDir         string     `mapstructure:"output_directory"`
@@ -255,6 +256,11 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 		b.config.DiskCompression = false
 	}
 
+	if b.config.UseBackingFile && !(b.config.DiskImage && b.config.Format == "qcow2") {
+		errs = packer.MultiErrorAppend(
+			errs, errors.New("use_backing_file can only be enabled for QCOW2 images and when disk_image is true"))
+	}
+
 	if _, ok := accels[b.config.Accelerator]; !ok {
 		errs = packer.MultiErrorAppend(
 			errs, errors.New("invalid accelerator, only 'kvm', 'tcg', 'xen', 'hax', 'hvf', or 'none' are allowed"))

builder/qemu/builder_test.go

@@ -224,6 +224,49 @@ func TestBuilderPrepare_Format(t *testing.T) {
 	}
 }
 
+func TestBuilderPrepare_UseBackingFile(t *testing.T) {
+	var b Builder
+	config := testConfig()
+
+	config["use_backing_file"] = true
+
+	// Bad: iso_url is not a disk_image
+	config["disk_image"] = false
+	config["format"] = "qcow2"
+	b = Builder{}
+	warns, err := b.Prepare(config)
+	if len(warns) > 0 {
+		t.Fatalf("bad: %#v", warns)
+	}
+	if err == nil {
+		t.Fatal("should have error")
+	}
+
+	// Bad: format is not 'qcow2'
+	config["disk_image"] = true
+	config["format"] = "raw"
+	b = Builder{}
+	warns, err = b.Prepare(config)
+	if len(warns) > 0 {
+		t.Fatalf("bad: %#v", warns)
+	}
+	if err == nil {
+		t.Fatal("should have error")
+	}
+
+	// Good: iso_url is a disk image and format is 'qcow2'
+	config["disk_image"] = true
+	config["format"] = "qcow2"
+	b = Builder{}
+	warns, err = b.Prepare(config)
+	if len(warns) > 0 {
+		t.Fatalf("bad: %#v", warns)
+	}
+	if err != nil {
+		t.Fatalf("should not have error: %s", err)
+	}
+}
+
 func TestBuilderPrepare_FloppyFiles(t *testing.T) {
 	var b Builder
 	config := testConfig()

builder/qemu/ssh.go

@@ -1,10 +1,15 @@
 package qemu
 
 import (
+	"fmt"
+	"net"
+	"os"
+
 	commonssh "github.com/hashicorp/packer/common/ssh"
-	"github.com/hashicorp/packer/communicator/ssh"
+	packerssh "github.com/hashicorp/packer/communicator/ssh"
 	"github.com/hashicorp/packer/helper/multistep"
 	gossh "golang.org/x/crypto/ssh"
+	"golang.org/x/crypto/ssh/agent"
 )
 
 func commHost(state multistep.StateBag) (string, error) {
@@ -19,10 +24,29 @@ func commPort(state multistep.StateBag) (int, error) {
 func sshConfig(state multistep.StateBag) (*gossh.ClientConfig, error) {
 	config := state.Get("config").(*Config)
 
-	auth := []gossh.AuthMethod{
+	var auth []gossh.AuthMethod
-		gossh.Password(config.Comm.SSHPassword),
+
-		gossh.KeyboardInteractive(
+	if config.Comm.SSHAgentAuth {
-			ssh.PasswordKeyboardInteractive(config.Comm.SSHPassword)),
+		authSock := os.Getenv("SSH_AUTH_SOCK")
+		if authSock == "" {
+			return nil, fmt.Errorf("SSH_AUTH_SOCK is not set")
+		}
+
+		sshAgent, err := net.Dial("unix", authSock)
+		if err != nil {
+			return nil, fmt.Errorf("Cannot connect to SSH Agent socket %q: %s", authSock, err)
+		}
+		auth = []gossh.AuthMethod{
+			gossh.PublicKeysCallback(agent.NewClient(sshAgent).Signers),
+		}
+	}
+
+	if config.Comm.SSHPassword != "" {
+		auth = append(auth,
+			gossh.Password(config.Comm.SSHPassword),
+			gossh.KeyboardInteractive(
+				packerssh.PasswordKeyboardInteractive(config.Comm.SSHPassword)),
+		)
 	}
 
 	if config.Comm.SSHPrivateKey != "" {

builder/qemu/step_convert_disk.go

@@ -4,7 +4,9 @@ import (
 	"context"
 	"fmt"
 	"path/filepath"
+	"strings"
 
+	"github.com/hashicorp/packer/common"
 	"github.com/hashicorp/packer/helper/multistep"
 	"github.com/hashicorp/packer/packer"
 
@@ -46,11 +48,32 @@ func (s *stepConvertDisk) Run(_ context.Context, state multistep.StateBag) multi
 	)
 
 	ui.Say("Converting hard drive...")
-	if err := driver.QemuImg(command...); err != nil {
+	// Retry the conversion a few times in case it takes the qemu process a
-		err := fmt.Errorf("Error converting hard drive: %s", err)
+	// moment to release the lock
-		state.Put("error", err)
+	err := common.Retry(1, 10, 10, func(_ uint) (bool, error) {
-		ui.Error(err.Error())
+		if err := driver.QemuImg(command...); err != nil {
-		return multistep.ActionHalt
+			if strings.Contains(err.Error(), `Failed to get shared "write" lock`) {
+				ui.Say("Error getting file lock for conversion; retrying...")
+				return false, nil
+			}
+			err = fmt.Errorf("Error converting hard drive: %s", err)
+			return true, err
+		}
+		return true, nil
+	})
+
+	if err != nil {
+		if err == common.RetryExhaustedError {
+			err = fmt.Errorf("Exhausted retries for getting file lock: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		} else {
+			err := fmt.Errorf("Error converting hard drive: %s", err)
+			state.Put("error", err)
+			ui.Error(err.Error())
+			return multistep.ActionHalt
+		}
 	}
 
 	if err := os.Rename(targetPath, sourcePath); err != nil {

builder/qemu/step_copy_disk.go

@@ -28,7 +28,7 @@ func (s *stepCopyDisk) Run(_ context.Context, state multistep.StateBag) multiste
 		path,
 	}
 
-	if config.DiskImage == false {
+	if !config.DiskImage || config.UseBackingFile {
 		return multistep.ActionContinue
 	}
 

builder/qemu/step_create_disk.go

@@ -23,11 +23,19 @@ func (s *stepCreateDisk) Run(_ context.Context, state multistep.StateBag) multis
 	command := []string{
 		"create",
 		"-f", config.Format,
-		path,
-		fmt.Sprintf("%vM", config.DiskSize),
 	}
 
-	if config.DiskImage == true {
+	if config.UseBackingFile {
+		isoPath := state.Get("iso_path").(string)
+		command = append(command, "-b", isoPath)
+	}
+
+	command = append(command,
+		path,
+		fmt.Sprintf("%vM", config.DiskSize),
+	)
+
+	if config.DiskImage && !config.UseBackingFile {
 		return multistep.ActionContinue
 	}
 

builder/qemu/step_type_boot_command.go

@@ -41,6 +41,7 @@ func (s *stepTypeBootCommand) Run(ctx context.Context, state multistep.StateBag)
 	httpPort := state.Get("http_port").(uint)
 	ui := state.Get("ui").(packer.Ui)
 	vncPort := state.Get("vnc_port").(uint)
+	vncIP := state.Get("vnc_ip").(string)
 
 	if config.VNCConfig.DisableVNC {
 		log.Println("Skipping boot command step...")
@@ -65,7 +66,7 @@ func (s *stepTypeBootCommand) Run(ctx context.Context, state multistep.StateBag)
 
 	// Connect to VNC
 	ui.Say("Connecting to VM via VNC")
-	nc, err := net.Dial("tcp", fmt.Sprintf("127.0.0.1:%d", vncPort))
+	nc, err := net.Dial("tcp", fmt.Sprintf("%s:%d", vncIP, vncPort))
 	if err != nil {
 		err := fmt.Errorf("Error connecting to VNC: %s", err)
 		state.Put("error", err)

builder/scaleway/config.go

@@ -29,6 +29,7 @@ type Config struct {
 	SnapshotName string `mapstructure:"snapshot_name"`
 	ImageName    string `mapstructure:"image_name"`
 	ServerName   string `mapstructure:"server_name"`
+	Bootscript   string `mapstructure:"bootscript"`
 
 	UserAgent string
 	ctx       interpolate.Context

builder/scaleway/step_create_server.go

@@ -20,9 +20,14 @@ func (s *stepCreateServer) Run(_ context.Context, state multistep.StateBag) mult
 	c := state.Get("config").(Config)
 	sshPubKey := state.Get("ssh_pubkey").(string)
 	tags := []string{}
+	var bootscript *string
 
 	ui.Say("Creating server...")
 
+	if c.Bootscript != "" {
+		bootscript = &c.Bootscript
+	}
+
 	if sshPubKey != "" {
 		tags = []string{fmt.Sprintf("AUTHORIZED_KEY=%s", strings.TrimSpace(sshPubKey))}
 	}
@@ -33,6 +38,7 @@ func (s *stepCreateServer) Run(_ context.Context, state multistep.StateBag) mult
 		Organization:   c.Organization,
 		CommercialType: c.CommercialType,
 		Tags:           tags,
+		Bootscript:     bootscript,
 	})
 
 	if err != nil {

builder/virtualbox/common/step_download_guest_additions.go

@@ -94,7 +94,7 @@ func (s *StepDownloadGuestAdditions) Run(ctx context.Context, state multistep.St
 		} else {
 			ui.Error(err.Error())
 			url = fmt.Sprintf(
-				"http://download.virtualbox.org/virtualbox/%s/%s",
+				"https://download.virtualbox.org/virtualbox/%s/%s",
 				version,
 				additionsName)
 		}
@@ -150,7 +150,7 @@ func (s *StepDownloadGuestAdditions) downloadAdditionsSHA256(ctx context.Context
 	// First things first, we get the list of checksums for the files available
 	// for this version.
 	checksumsUrl := fmt.Sprintf(
-		"http://download.virtualbox.org/virtualbox/%s/SHA256SUMS",
+		"https://download.virtualbox.org/virtualbox/%s/SHA256SUMS",
 		additionsVersion)
 
 	checksumsFile, err := ioutil.TempFile("", "packer")

builder/virtualbox/iso/step_attach_iso.go

@@ -3,6 +3,7 @@ package iso
 import (
 	"context"
 	"fmt"
+	"path/filepath"
 
 	vboxcommon "github.com/hashicorp/packer/builder/virtualbox/common"
 	"github.com/hashicorp/packer/helper/multistep"
@@ -34,6 +35,16 @@ func (s *stepAttachISO) Run(_ context.Context, state multistep.StateBag) multist
 		device = "0"
 	}
 
+	// If it's a symlink, resolve it to it's target.
+	resolvedIsoPath, err := filepath.EvalSymlinks(isoPath)
+	if err != nil {
+		err := fmt.Errorf("Error resolving symlink for ISO: %s", err)
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	}
+	isoPath = resolvedIsoPath
+
 	// Attach the disk to the controller
 	command := []string{
 		"storageattach", vmName,

builder/vmware/common/driver.go

@@ -23,7 +23,7 @@ type Driver interface {
 	// Clone clones the VMX and the disk to the destination path. The
 	// destination is a path to the VMX file. The disk will be copied
 	// to that same directory.
-	Clone(dst string, src string) error
+	Clone(dst string, src string, cloneType bool) error
 
 	// CompactDisk compacts a virtual disk.
 	CompactDisk(string) error
@@ -358,7 +358,8 @@ func (d *VmwareDriver) GuestIP(state multistep.StateBag) (string, error) {
 		// open up the lease and read its contents
 		fh, err := os.Open(dhcpLeasesPath)
 		if err != nil {
-			return "", err
+			log.Printf("Error while reading DHCP lease path file %s: %s", dhcpLeasesPath, err.Error())
+			continue
 		}
 		defer fh.Close()
 

builder/vmware/common/driver_fusion5.go

@@ -24,7 +24,7 @@ type Fusion5Driver struct {
 	SSHConfig *SSHConfig
 }
 
-func (d *Fusion5Driver) Clone(dst, src string) error {
+func (d *Fusion5Driver) Clone(dst, src string, linked bool) error {
 	return errors.New("Cloning is not supported with Fusion 5. Please use Fusion 6+.")
 }
 

builder/vmware/common/driver_fusion6.go

@@ -18,11 +18,19 @@ type Fusion6Driver struct {
 	Fusion5Driver
 }
 
-func (d *Fusion6Driver) Clone(dst, src string) error {
+func (d *Fusion6Driver) Clone(dst, src string, linked bool) error {
+
+	var cloneType string
+	if linked {
+		cloneType = "linked"
+	} else {
+		cloneType = "full"
+	}
+
 	cmd := exec.Command(d.vmrunPath(),
 		"-T", "fusion",
 		"clone", src, dst,
-		"full")
+		cloneType)
 	if _, _, err := runAndLog(cmd); err != nil {
 		if strings.Contains(err.Error(), "parameters was invalid") {
 			return fmt.Errorf(

builder/vmware/common/driver_mock.go

@@ -13,6 +13,7 @@ type DriverMock struct {
 	CloneCalled bool
 	CloneDst    string
 	CloneSrc    string
+	Linked      bool
 	CloneErr    error
 
 	CompactDiskCalled bool
@@ -107,10 +108,11 @@ func (m NetworkMapperMock) DeviceIntoName(device string) (string, error) {
 	return "", nil
 }
 
-func (d *DriverMock) Clone(dst string, src string) error {
+func (d *DriverMock) Clone(dst string, src string, linked bool) error {
 	d.CloneCalled = true
 	d.CloneDst = dst
 	d.CloneSrc = src
+	d.Linked = linked
 	return d.CloneErr
 }
 

builder/vmware/common/driver_player5.go

@@ -25,7 +25,7 @@ type Player5Driver struct {
 	SSHConfig *SSHConfig
 }
 
-func (d *Player5Driver) Clone(dst, src string) error {
+func (d *Player5Driver) Clone(dst, src string, linked bool) error {
 	return errors.New("Cloning is not supported with VMWare Player version 5. Please use VMWare Player version 6, or greater.")
 }
 

builder/vmware/common/driver_player6.go

@@ -13,13 +13,20 @@ type Player6Driver struct {
 	Player5Driver
 }
 
-func (d *Player6Driver) Clone(dst, src string) error {
+func (d *Player6Driver) Clone(dst, src string, linked bool) error {
 	// TODO(rasa) check if running player+, not just player
 
+	var cloneType string
+	if linked {
+		cloneType = "linked"
+	} else {
+		cloneType = "full"
+	}
+
 	cmd := exec.Command(d.Player5Driver.VmrunPath,
 		"-T", "ws",
 		"clone", src, dst,
-		"full")
+		cloneType)
 
 	if _, _, err := runAndLog(cmd); err != nil {
 		return err

builder/vmware/common/driver_workstation10.go

@@ -13,11 +13,19 @@ type Workstation10Driver struct {
 	Workstation9Driver
 }
 
-func (d *Workstation10Driver) Clone(dst, src string) error {
+func (d *Workstation10Driver) Clone(dst, src string, linked bool) error {
+
+	var cloneType string
+	if linked {
+		cloneType = "linked"
+	} else {
+		cloneType = "full"
+	}
+
 	cmd := exec.Command(d.Workstation9Driver.VmrunPath,
 		"-T", "ws",
 		"clone", src, dst,
-		"full")
+		cloneType)
 
 	if _, _, err := runAndLog(cmd); err != nil {
 		return err

builder/vmware/common/driver_workstation9.go

@@ -24,7 +24,7 @@ type Workstation9Driver struct {
 	SSHConfig *SSHConfig
 }
 
-func (d *Workstation9Driver) Clone(dst, src string) error {
+func (d *Workstation9Driver) Clone(dst, src string, linked bool) error {
 	return errors.New("Cloning is not supported with VMware WS version 9. Please use VMware WS version 10, or greater.")
 }
 

builder/vmware/common/step_configure_vmx.go

@@ -16,6 +16,9 @@ import (
 //
 // Uses:
 //   vmx_path string
+//
+// Produces:
+//   display_name string - Value of the displayName key set in the VMX file
 type StepConfigureVMX struct {
 	CustomData map[string]string
 	SkipFloppy bool
@@ -73,6 +76,19 @@ func (s *StepConfigureVMX) Run(_ context.Context, state multistep.StateBag) mult
 		return multistep.ActionHalt
 	}
 
+	// If the build is taking place on a remote ESX server, the displayName
+	// will be needed for discovery of the VM's IP address and for export
+	// of the VM. The displayName key should always be set in the VMX file,
+	// so error if we don't find it
+	if displayName, ok := vmxData["displayname"]; !ok { // Packer converts key names to lowercase!
+		err := fmt.Errorf("Error: Could not get value of displayName from VMX data")
+		state.Put("error", err)
+		ui.Error(err.Error())
+		return multistep.ActionHalt
+	} else {
+		state.Put("display_name", displayName)
+	}
+
 	return multistep.ActionContinue
 }
 

builder/vmware/common/step_configure_vmx_test.go

@@ -14,6 +14,9 @@ func testVMXFile(t *testing.T) string {
 	if err != nil {
 		t.Fatalf("err: %s", err)
 	}
+
+	// displayName must always be set
+	err = WriteVMX(tf.Name(), map[string]string{"displayName": "PackerBuild"})
 	tf.Close()
 
 	return tf.Name()
@@ -132,12 +135,29 @@ func TestStepConfigureVMX_generatedAddresses(t *testing.T) {
 	vmxPath := testVMXFile(t)
 	defer os.Remove(vmxPath)
 
-	err := WriteVMX(vmxPath, map[string]string{
+	additionalTestVmxData := []struct {
-		"foo": "bar",
+		Key   string
-		"ethernet0.generatedAddress":       "foo",
+		Value string
-		"ethernet1.generatedAddress":       "foo",
+	}{
-		"ethernet1.generatedAddressOffset": "foo",
+		{"foo", "bar"},
-	})
+		{"ethernet0.generatedaddress", "foo"},
+		{"ethernet1.generatedaddress", "foo"},
+		{"ethernet1.generatedaddressoffset", "foo"},
+	}
+
+	// Get any existing VMX data from the VMX file
+	vmxData, err := ReadVMX(vmxPath)
+	if err != nil {
+		t.Fatalf("err %s", err)
+	}
+
+	// Add the additional key/value pairs we need for this test to the existing VMX data
+	for _, data := range additionalTestVmxData {
+		vmxData[data.Key] = data.Value
+	}
+
+	// Recreate the VMX file so it includes all the data needed for this test
+	err = WriteVMX(vmxPath, vmxData)
 	if err != nil {
 		t.Fatalf("err: %s", err)
 	}
@@ -157,7 +177,7 @@ func TestStepConfigureVMX_generatedAddresses(t *testing.T) {
 	if err != nil {
 		t.Fatalf("err: %s", err)
 	}
-	vmxData := ParseVMX(string(vmxContents))
+	vmxData = ParseVMX(string(vmxContents))
 
 	cases := []struct {
 		Key   string
@@ -180,5 +200,71 @@ func TestStepConfigureVMX_generatedAddresses(t *testing.T) {
 			}
 		}
 	}
+}
+
+// Should fail if the displayName key is not found in the VMX
+func TestStepConfigureVMX_displayNameMissing(t *testing.T) {
+	state := testState(t)
+	step := new(StepConfigureVMX)
+
+	// testVMXFile adds displayName key/value pair to the VMX
+	vmxPath := testVMXFile(t)
+	defer os.Remove(vmxPath)
+
+	// Bad: Delete displayName from the VMX/Create an empty VMX file
+	err := WriteVMX(vmxPath, map[string]string{})
+	if err != nil {
+		t.Fatalf("err: %s", err)
+	}
+
+	state.Put("vmx_path", vmxPath)
+
+	// Test the run
+	if action := step.Run(context.Background(), state); action != multistep.ActionHalt {
+		t.Fatalf("bad action: %#v. Should halt when displayName key is missing from VMX", action)
+	}
+	if _, ok := state.GetOk("error"); !ok {
+		t.Fatal("should store error in state when displayName key is missing from VMX")
+	}
+}
+
+// Should store the value of displayName in the statebag
+func TestStepConfigureVMX_displayNameStore(t *testing.T) {
+	state := testState(t)
+	step := new(StepConfigureVMX)
+
+	// testVMXFile adds displayName key/value pair to the VMX
+	vmxPath := testVMXFile(t)
+	defer os.Remove(vmxPath)
+
+	state.Put("vmx_path", vmxPath)
+
+	// Test the run
+	if action := step.Run(context.Background(), state); action != multistep.ActionContinue {
+		t.Fatalf("bad action: %#v", action)
+	}
+	if _, ok := state.GetOk("error"); ok {
+		t.Fatal("should NOT have error")
+	}
+
+	// The value of displayName must be stored in the statebag
+	if _, ok := state.GetOk("display_name"); !ok {
+		t.Fatalf("displayName should be stored in the statebag as 'display_name'")
+	}
+}
+
+func TestStepConfigureVMX_vmxPathBad(t *testing.T) {
+	state := testState(t)
+	step := new(StepConfigureVMX)
+
+	state.Put("vmx_path", "some_bad_path")
+
+	// Test the run
+	if action := step.Run(context.Background(), state); action != multistep.ActionHalt {
+		t.Fatalf("bad action: %#v. Should halt when vmxPath is bad", action)
+	}
+	if _, ok := state.GetOk("error"); !ok {
+		t.Fatal("should store error in state when vmxPath is bad")
+	}
 
 }

builder/vmware/iso/builder.go

@@ -221,6 +221,11 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, error) {
 			fmt.Errorf("format must be one of ova, ovf, or vmx"))
 	}
 
+	if b.config.RemoteType == "esx5" && b.config.SkipExport != true && b.config.RemotePassword == "" {
+		errs = packer.MultiErrorAppend(errs,
+			fmt.Errorf("exporting the vm (with ovftool) requires that you set a value for remote_password"))
+	}
+
 	// Warnings
 	if b.config.ShutdownCommand == "" {
 		warnings = append(warnings,

builder/vmware/iso/builder_test.go

@@ -145,6 +145,7 @@ func TestBuilderPrepare_RemoteType(t *testing.T) {
 
 	config["format"] = "ovf"
 	config["remote_host"] = "foobar.example.com"
+	config["remote_password"] = "supersecret"
 	// Bad
 	config["remote_type"] = "foobar"
 	warns, err := b.Prepare(config)
@@ -155,9 +156,10 @@ func TestBuilderPrepare_RemoteType(t *testing.T) {
 		t.Fatal("should have error")
 	}
 
-	config["remote_host"] = ""
+	config["remote_type"] = "esx5"
-	config["remote_type"] = ""
 	// Bad
+	config["remote_host"] = ""
+	b = Builder{}
 	warns, err = b.Prepare(config)
 	if len(warns) > 0 {
 		t.Fatalf("bad: %#v", warns)
@@ -167,8 +169,10 @@ func TestBuilderPrepare_RemoteType(t *testing.T) {
 	}
 
 	// Good
-	config["remote_type"] = "esx5"
+	config["remote_type"] = ""
-	config["remote_host"] = "foobar.example.com"
+	config["remote_host"] = ""
+	config["remote_password"] = ""
+	config["remote_private_key_file"] = ""
 	b = Builder{}
 	warns, err = b.Prepare(config)
 	if len(warns) > 0 {
@@ -181,6 +185,7 @@ func TestBuilderPrepare_RemoteType(t *testing.T) {
 	// Good
 	config["remote_type"] = "esx5"
 	config["remote_host"] = "foobar.example.com"
+	config["remote_password"] = "supersecret"
 	b = Builder{}
 	warns, err = b.Prepare(config)
 	if len(warns) > 0 {
@@ -191,6 +196,34 @@ func TestBuilderPrepare_RemoteType(t *testing.T) {
 	}
 }
 
+func TestBuilderPrepare_RemoteExport(t *testing.T) {
+	var b Builder
+	config := testConfig()
+
+	config["remote_type"] = "esx5"
+	config["remote_host"] = "foobar.example.com"
+	// Bad
+	config["remote_password"] = ""
+	warns, err := b.Prepare(config)
+	if len(warns) != 0 {
+		t.Fatalf("bad: %#v", warns)
+	}
+	if err == nil {
+		t.Fatal("should have error")
+	}
+
+	// Good
+	config["remote_password"] = "supersecret"
+	b = Builder{}
+	warns, err = b.Prepare(config)
+	if len(warns) != 0 {
+		t.Fatalf("err: %s", err)
+	}
+	if err != nil {
+		t.Fatalf("should not have error: %s", err)
+	}
+}
+
 func TestBuilderPrepare_Format(t *testing.T) {
 	var b Builder
 	config := testConfig()

builder/vmware/iso/driver_esx5.go

@@ -41,7 +41,7 @@ type ESX5Driver struct {
 	vmId      string
 }
 
-func (d *ESX5Driver) Clone(dst, src string) error {
+func (d *ESX5Driver) Clone(dst, src string, linked bool) error {
 	return errors.New("Cloning is not supported with the ESX driver.")
 }
 
@@ -389,7 +389,13 @@ func (d *ESX5Driver) CommHost(state multistep.StateBag) (string, error) {
 		return "", err
 	}
 
-	record, err := r.find("Name", config.VMName)
+	// The value in the Name field returned by 'esxcli network vm list'
+	// corresponds directly to the value of displayName set in the VMX file
+	var displayName string
+	if v, ok := state.GetOk("display_name"); ok {
+		displayName = v.(string)
+	}
+	record, err := r.find("Name", displayName)
 	if err != nil {
 		return "", err
 	}

builder/vmware/iso/step_export.go

@@ -14,13 +14,17 @@ import (
 	"github.com/hashicorp/packer/packer"
 )
 
+// This step exports a VM built on ESXi using ovftool
+//
+// Uses:
+//   display_name string
 type StepExport struct {
 	Format     string
 	SkipExport bool
 	OutputDir  string
 }
 
-func (s *StepExport) generateArgs(c *Config, hidePassword bool) []string {
+func (s *StepExport) generateArgs(c *Config, displayName string, hidePassword bool) []string {
 	password := url.QueryEscape(c.RemotePassword)
 	if hidePassword {
 		password = "****"
@@ -29,7 +33,7 @@ func (s *StepExport) generateArgs(c *Config, hidePassword bool) []string {
 		"--noSSLVerify=true",
 		"--skipManifestCheck",
 		"-tt=" + s.Format,
-		"vi://" + c.RemoteUser + ":" + password + "@" + c.RemoteHost + "/" + c.VMName,
+		"vi://" + c.RemoteUser + ":" + password + "@" + c.RemoteHost + "/" + displayName,
 		s.OutputDir,
 	}
 	return append(c.OVFToolOptions, args...)
@@ -72,9 +76,13 @@ func (s *StepExport) Run(_ context.Context, state multistep.StateBag) multistep.
 	}
 
 	ui.Say("Exporting virtual machine...")
-	ui.Message(fmt.Sprintf("Executing: %s %s", ovftool, strings.Join(s.generateArgs(c, true), " ")))
+	var displayName string
+	if v, ok := state.GetOk("display_name"); ok {
+		displayName = v.(string)
+	}
+	ui.Message(fmt.Sprintf("Executing: %s %s", ovftool, strings.Join(s.generateArgs(c, displayName, true), " ")))
 	var out bytes.Buffer
-	cmd := exec.Command(ovftool, s.generateArgs(c, false)...)
+	cmd := exec.Command(ovftool, s.generateArgs(c, displayName, false)...)
 	cmd.Stdout = &out
 	if err := cmd.Run(); err != nil {
 		err := fmt.Errorf("Error exporting virtual machine: %s\n%s\n", err, out.String())