| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-12 | |||
| 21:59:32 | brandor5 | haha | |
| 21:59:46 | brandor5 | i'll keep poking then... double check my headers and whatnot | |
| 21:59:55 | brandor5 | thanks for the help | |
| 22:00:09 | mriedem | are you polling on the 'status' field in the server show response or something like vm_state? | |
| 22:01:08 | brandor5 | https://pastebin.com/ijVe0J42 | |
| 22:01:31 | melwitt | mriedem: what's the trick to bfv with more than one --block-device? I keep getting ERROR state saying 'no image supplied' even though the first one I give is bootindex=0 with source=image | |
| 22:02:06 | mriedem | brandor5: that looks ok | |
| 22:02:35 | mriedem | melwitt: are you trying to attach data volumes or have nova create the volumes? | |
| 22:02:54 | mriedem | it's probably easiest to just boot and attach 2 data volumes (so not boot_index=0) | |
| 22:03:08 | melwitt | mriedem: I'm consistently trying to have nova create the boot volume, and attach a non-bootable volume I created in cinder ahead | |
| 22:03:09 | mriedem | i.e. pre-existing volumes | |
| 22:03:27 | mriedem | what is your source_type and destination_type? | |
| 22:04:07 | melwitt | this was my latest attempt http://paste.openstack.org/show/737169/ | |
| 22:04:32 | mriedem | --block-device source=image,id=d7b0d7cb-1373-456d-bcec-29fbec2f84e1,dest=volume,size=1,shutdown=preserve,bootindex=0 | |
| 22:04:35 | mriedem | is that id an image id? | |
| 22:04:45 | melwitt | yes | |
| 22:04:51 | mriedem | --block-device source=volume,id=33550efe-6a7b-4956-a7e0-5bd50004ec30,dest=volume,size=1,shutdown=preserve | |
| 22:04:54 | mriedem | and that id is a volume id? | |
| 22:04:57 | melwitt | and that one is a volume id | |
| 22:05:13 | mriedem | you don't need to specify size on the latter, but nova should ignore that | |
| 22:05:36 | melwitt | I think it can be done but I don't know how. I think I've seen launchpad bug reports where people were booting with multiple volumes | |
| 22:06:09 | mriedem | where are you seeing the "no image supplied"? | |
| 22:06:27 | melwitt | in the nova show fault field | |
| 22:06:32 | mriedem | http://codesearch.openstack.org/?q=no%20image%20supplied&i=nope&files=&repos= | |
| 22:06:40 | mriedem | that's not in the code that i can see | |
| 22:06:43 | mriedem | besides the show response | |
| 22:06:48 | mriedem | which is normal | |
| 22:06:53 | mriedem | paste your fault | |
| 22:07:40 | melwitt | http://paste.openstack.org/show/737170/ | |
| 22:07:56 | melwitt | oh sorry | |
| 22:07:57 | melwitt | dammmit | |
| 22:08:05 | melwitt | that was the image field | |
| 22:08:34 | Kevin_Zheng | mriedem: just shelve the vm and all the volume will be in reserved status and cannot be detached | |
| 22:08:36 | mriedem | yeah that's normal | |
| 22:08:37 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Handle tags in _bury_in_cell0 https://review.openstack.org/624813 | |
| 22:08:40 | mriedem | b/c there is no image ref if you bfv | |
| 22:08:59 | mriedem | Kevin_Zheng: we have a tempest test that does exactly that | |
| 22:09:31 | melwitt | yeah, I am so rusty on this. that's embarrassing | |
| 22:09:39 | Kevin_Zheng | I didn’t check yet but that’s the result I got | |
| 22:09:51 | mriedem | good thing i don't have a twitter account to let everyone know about this | |
| 22:09:52 | Kevin_Zheng | And also the code does it | |
| 22:09:56 | melwitt | I was thinking that sentence was the fault field, but the fault field wasn't even showing because I wasn't authed as admin. lots of good stuff going on here | |
| 22:10:20 | melwitt | haha yup | |
| 22:10:40 | mriedem | Kevin_Zheng: is this vanilla devstack or something with your detach boot volume changes applied? | |
| 22:10:47 | melwitt | failure to prep block device, which is probably my conf limit code firing | |
| 22:11:12 | mriedem | melwitt: yes probably, and we don't reschedule on failures from _prep_block_device i don't think | |
| 22:11:19 | mriedem | so we just crap out | |
| 22:11:46 | brandor5 | mriedem: i found the problem, the resize call was firing too fast... I put a "sleep" in for a few seconds and it worked | |
| 22:11:51 | Kevin_Zheng | DevStack | |
| 22:12:27 | mriedem | brandor5: hmm, that's weird, because as soon as the server status is VERIFY_RESIZE is should be ok to call confirmResize | |
| 22:12:35 | mriedem | especially because i'm sure tempest does exactly the same thing, | |
| 22:12:42 | mriedem | poll for VERIFY_RESIZE and then immediately confirm | |
| 22:13:02 | brandor5 | yeah, that's what I'm doing | |
| 22:13:12 | brandor5 | if I sleep for 3 seconds after that it works | |
| 22:13:46 | Kevin_Zheng | I did not change the shelve flow, I only change the detach attach flow and it will not affect as it was blocked in the very early stage calling begin-detaching | |
| 22:13:57 | brandor5 | unless my polling isn't working the way I think | |
| 22:14:02 | brandor5 | I'll double check that really quick | |
| 22:14:57 | mriedem | this is where nova-compute changes the status https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3943 | |
| 22:15:16 | mriedem | this is the api code for the status field https://github.com/openstack/nova/blob/newton-eol/nova/api/openstack/common.py#L79 | |
| 22:15:32 | mriedem | so once compute does the update to the db, and the api returns VERIFY_RESIZE is should be ok to confirm | |
| 22:15:52 | mriedem | Kevin_Zheng: did you look at the tempest test i pointed out? | |
| 22:16:40 | Kevin_Zheng | Not yet, I will check it today | |
| 22:16:52 | mriedem | good morning btw | |
| 22:17:03 | mriedem | 6am what are you doing | |
| 22:17:35 | Kevin_Zheng | My kid fall down the bed so everyone is up:) | |
| 22:18:35 | mriedem | ha | |
| 22:18:39 | mriedem | time for guard rails | |
| 22:18:46 | mriedem | or strapts | |
| 22:18:47 | mriedem | *straps | |
| 22:18:54 | brandor5 | mriedem: found the problem... it was an error on my part... if you look at the paste I linked the register,until, retries, and delay options were indented incorrectly and not being honored....... so it was just running the cehck once and then moving on through | |
| 22:19:26 | mriedem | brandor5: ah ok | |
| 22:19:33 | mriedem | i knew it couldn't be nova's fault, nova is never at fault | |
| 22:19:38 | brandor5 | hehe | |
| 22:19:44 | Kevin_Zheng | :) go back to sleep now | |
| 22:19:45 | brandor5 | it's dns | |
| 22:19:45 | mriedem | it probably worked in the cli just b/c of a time delay | |
| 22:19:51 | brandor5 | or the network | |
| 22:19:53 | mriedem | building the client and such | |
| 22:19:54 | mriedem | or that | |
| 22:19:54 | brandor5 | those are my favorits :D | |
| 22:21:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle tags in _bury_in_cell0 https://review.openstack.org/624824 | |
| 22:22:39 | melwitt | confirmed, it was my limit code. so it does work for bfv. and it didn't create a new volume before it failed. I will test it more, maybe it's too good to be true | |
| 22:22:57 | mriedem | we should get that fault message cleaned up to actually have the error message in it | |
| 22:23:02 | melwitt | aye | |
| 22:23:22 | mriedem | could have sworn i did something in that code for that, but might have been logging | |
| 22:24:02 | mriedem | yeah https://review.openstack.org/#/q/I20a5e8e5e10dd505c1b24c208f919c6550e9d1a4 | |
| 22:25:02 | mriedem | maybe "Failure prepping block device" is just your error message | |
| 22:25:50 | melwitt | indeed in the log: Failure prepping block device: TooManyDiskDevices: The maximum allowed number of disk devices to attach to a single instance has bee | |
| 22:25:50 | melwitt | n exceeded | |
| 22:26:08 | melwitt | le sigh, newline | |
| 22:36:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Handle tags in _bury_in_cell0 https://review.openstack.org/624825 | |
| 22:48:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Replace ThreadPoolExecutor with GreenThreadPoolExecutor https://review.openstack.org/624740 | |
| 22:48:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add python 3.7 unit and functional tox jobs https://review.openstack.org/624055 | |
| 22:48:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add py36/py37 functional jobs to the experimental queue https://review.openstack.org/624826 | |
| 22:48:23 | melwitt | found that the error message gets squashed here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2341-L2343 and that's what ends up in the instance fault | |
| 22:49:39 | melwitt | feels like a separate patch for improving the information there, I can try it and see how it looks, going to be a much longer string | |
| 23:15:53 | openstackgerrit | Matt Riedemann proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | |
| 23:15:54 | openstackgerrit | Matt Riedemann proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | |
| 23:21:44 | mriedem | melwitt: oh ok, we should just do similar in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2341 then and probably include the underlying reason | |
| 23:21:54 | mriedem | also, duh duh duh nova-lvm job is busted (privsep) http://logs.openstack.org/43/619143/12/check/nova-lvm/786180c/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_12_35_39_607002 | |