Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-12
21:51:27 brandor5 mriedem: oh cool, thanks... any idea on the error I'm seeing?
21:51:38 brandor5 especially that it works when I use the command line client
21:52:18 mriedem can't confirm a resize on a stopped vm in newton https://github.com/openstack/nova/blob/newton-eol/nova/compute/api.py#L2924
21:52:38 mriedem nor master https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3489
21:53:19 brandor5 hmm that's weird because `nova resize-confirm <instance id>` definitely works
21:53:32 mriedem works works? or just doesn't say it failed?
21:53:38 brandor5 works works
21:53:40 mriedem which version of novaclient?
21:53:56 brandor5 10.1.0
21:54:29 brandor5 and when I run nova in debug I've verified that I'm running the same 'curl' that novaclient is running
21:55:05 mriedem no magic there https://github.com/openstack/python-novaclient/blob/10.1.0/novaclient/v2/servers.py#L1533
21:55:44 mriedem can you show the server status before doing each (cli and curl) to verify?
21:56:10 brandor5 yeah, I'm doing that... I have a watch running `openstack server show <instance_id>`
21:56:29 brandor5 if I try to hit the api directly I get a 409
21:56:36 mriedem so you have some script or something that's just auto-confirming a resize as soon as the instance status is VERIFY_RESIZE?
21:56:37 brandor5 if I use novaclient it works
21:56:44 brandor5 correct
21:56:56 mriedem there is a config option for auto-confirming migrations
21:56:58 mriedem ..
21:57:33 brandor5 yeah, that's across the board and the docs say it's not recommended... my script is just for putting a compute into 'maintenance' mode
21:58:22 brandor5 It's actually ansible playbook that sets maintenance mode, checks for errored instances, manually moves shutoff instances, and then blasts the active instances'
21:59:11 brandor5 unfortunately right now I can't use the command line clients because ansible tower doesn't have them installed... (that's a ticket i have open with redhat)
21:59:21 mriedem gd red hat
21:59:31 mriedem i can't see anything that would be a difference between the client and straight curl
21:59:32 brandor5 haha
21:59:46 brandor5 i'll keep poking then... double check my headers and whatnot
21:59:55 brandor5 thanks for the help
22:00:09 mriedem are you polling on the 'status' field in the server show response or something like vm_state?
22:01:08 brandor5 https://pastebin.com/ijVe0J42
22:01:31 melwitt mriedem: what's the trick to bfv with more than one --block-device? I keep getting ERROR state saying 'no image supplied' even though the first one I give is bootindex=0 with source=image
22:02:06 mriedem brandor5: that looks ok
22:02:35 mriedem melwitt: are you trying to attach data volumes or have nova create the volumes?
22:02:54 mriedem it's probably easiest to just boot and attach 2 data volumes (so not boot_index=0)
22:03:08 melwitt mriedem: I'm consistently trying to have nova create the boot volume, and attach a non-bootable volume I created in cinder ahead
22:03:09 mriedem i.e. pre-existing volumes
22:03:27 mriedem what is your source_type and destination_type?
22:04:07 melwitt this was my latest attempt http://paste.openstack.org/show/737169/
22:04:32 mriedem --block-device source=image,id=d7b0d7cb-1373-456d-bcec-29fbec2f84e1,dest=volume,size=1,shutdown=preserve,bootindex=0
22:04:35 mriedem is that id an image id?
22:04:45 melwitt yes
22:04:51 mriedem --block-device source=volume,id=33550efe-6a7b-4956-a7e0-5bd50004ec30,dest=volume,size=1,shutdown=preserve
22:04:54 mriedem and that id is a volume id?
22:04:57 melwitt and that one is a volume id
22:05:13 mriedem you don't need to specify size on the latter, but nova should ignore that
22:05:36 melwitt I think it can be done but I don't know how. I think I've seen launchpad bug reports where people were booting with multiple volumes
22:06:09 mriedem where are you seeing the "no image supplied"?
22:06:27 melwitt in the nova show fault field
22:06:32 mriedem http://codesearch.openstack.org/?q=no%20image%20supplied&i=nope&files=&repos=
22:06:40 mriedem that's not in the code that i can see
22:06:43 mriedem besides the show response
22:06:48 mriedem which is normal
22:06:53 mriedem paste your fault
22:07:40 melwitt http://paste.openstack.org/show/737170/
22:07:56 melwitt oh sorry
22:07:57 melwitt dammmit
22:08:05 melwitt that was the image field
22:08:34 Kevin_Zheng mriedem: just shelve the vm and all the volume will be in reserved status and cannot be detached
22:08:36 mriedem yeah that's normal
22:08:37 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813
22:08:40 mriedem b/c there is no image ref if you bfv
22:08:59 mriedem Kevin_Zheng: we have a tempest test that does exactly that
22:09:31 melwitt yeah, I am so rusty on this. that's embarrassing
22:09:39 Kevin_Zheng I didn’t check yet but that’s the result I got
22:09:51 mriedem good thing i don't have a twitter account to let everyone know about this
22:09:52 Kevin_Zheng And also the code does it
22:09:56 melwitt I was thinking that sentence was the fault field, but the fault field wasn't even showing because I wasn't authed as admin. lots of good stuff going on here
22:10:20 melwitt haha yup
22:10:40 mriedem Kevin_Zheng: is this vanilla devstack or something with your detach boot volume changes applied?
22:10:47 melwitt failure to prep block device, which is probably my conf limit code firing
22:11:12 mriedem melwitt: yes probably, and we don't reschedule on failures from _prep_block_device i don't think
22:11:19 mriedem so we just crap out
22:11:46 brandor5 mriedem: i found the problem, the resize call was firing too fast... I put a "sleep" in for a few seconds and it worked
22:11:51 Kevin_Zheng DevStack
22:12:27 mriedem brandor5: hmm, that's weird, because as soon as the server status is VERIFY_RESIZE is should be ok to call confirmResize
22:12:35 mriedem especially because i'm sure tempest does exactly the same thing,
22:12:42 mriedem poll for VERIFY_RESIZE and then immediately confirm
22:13:02 brandor5 yeah, that's what I'm doing
22:13:12 brandor5 if I sleep for 3 seconds after that it works
22:13:46 Kevin_Zheng I did not change the shelve flow, I only change the detach attach flow and it will not affect as it was blocked in the very early stage calling begin-detaching
22:13:57 brandor5 unless my polling isn't working the way I think
22:14:02 brandor5 I'll double check that really quick
22:14:57 mriedem this is where nova-compute changes the status https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3943
22:15:16 mriedem this is the api code for the status field https://github.com/openstack/nova/blob/newton-eol/nova/api/openstack/common.py#L79
22:15:32 mriedem so once compute does the update to the db, and the api returns VERIFY_RESIZE is should be ok to confirm
22:15:52 mriedem Kevin_Zheng: did you look at the tempest test i pointed out?
22:16:40 Kevin_Zheng Not yet, I will check it today
22:16:52 mriedem good morning btw
22:17:03 mriedem 6am what are you doing
22:17:35 Kevin_Zheng My kid fall down the bed so everyone is up:)
22:18:35 mriedem ha
22:18:39 mriedem time for guard rails
22:18:46 mriedem or strapts
22:18:47 mriedem *straps
22:18:54 brandor5 mriedem: found the problem... it was an error on my part... if you look at the paste I linked the register,until, retries, and delay options were indented incorrectly and not being honored....... so it was just running the cehck once and then moving on through
22:19:26 mriedem brandor5: ah ok
22:19:33 mriedem i knew it couldn't be nova's fault, nova is never at fault
22:19:38 brandor5 hehe
22:19:44 Kevin_Zheng :) go back to sleep now
22:19:45 mriedem it probably worked in the cli just b/c of a time delay
22:19:45 brandor5 it's dns

Earlier   Later