Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-04
15:14:45 kashyap Damn, since I 'discovered' the bug, I get the pleasure of filing the Blueprint I guess :P
15:14:58 kashyap sean-k-mooney: More seriously, paperwork question: does this require a spec? Since it's user-impacting?
15:15:29 kashyap Or a spec-less blueprint is reasonable enough?
15:16:41 sean-k-mooney kashyap: spec-less blueprint. extra specs are a gray area as they are not technical part of the versioned api but are user fasing so mriedem or someelse will likely comment on the blueprint if a specs is needed
15:17:18 prometheanfire stephenfin: https://gist.githubusercontent.com/prometheanfire/6512134e799ec8c08c3f080150f60d19/raw/7cdb3a9a350c19540b1c930077e164786226636b/gistfile1.txt
15:17:20 kashyap sean-k-mooney: Yep, noted; thanks for the discussion.
15:18:25 sean-k-mooney by the way i have been using http://codesearch.openstack.org/ a lot more recently instead of using github to such for these things its pretty good
15:19:07 kashyap Ah, nice.
15:19:23 kashyap sean-k-mooney: BTW, seems like "hw:machine_type" isn't documented a flavor extra specs here: https://docs.openstack.org/nova/latest/user/flavors.html
15:19:33 kashyap Is that so?
15:20:01 sean-k-mooney kashyap: correct it is documented here https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L34-L38
15:20:44 sean-k-mooney accouding to the glace metadef registry its only valid on image not flavor
15:21:22 kashyap sean-k-mooney: Hmm, but the syntax is slightly different: hw:machine_type vs hw_machine_type (in nova.conf)
15:21:37 kashyap Right, this seem to work: `openstack flavor set --property hw:machine_type=x86_64=q35 test.q35`
15:21:41 stephenfin prometheanfire: Based on that, it seems oslo.service package in your virtualenv is starting a thread using the system oslo.service package. I've no idea why that would happen
15:21:59 stephenfin prometheanfire: Might be worth asking on #openstack-oslo to see if anyone else has seen this before
15:22:11 sean-k-mooney kashyap: hw: is the namesapced flavor syntax, images dont have namespaces so the namespace is prepended with an _ instead of :
15:22:40 sean-k-mooney kashyap: https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L7-L11 tell you these are only valid in the image
15:22:44 stephenfin prometheanfire: You've patched 'usr/lib64/python3.5/site-packages/nova', I assume?
15:22:49 kashyap sean-k-mooney: Nod; I'll send a doc patch to document it here as well: https://docs.openstack.org/nova/latest/user/flavors.html
15:23:23 sean-k-mooney kashyap: if its valid in both it looks like https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L16
15:23:25 prometheanfire stephenfin: that file didn't exist at the time :P
15:23:39 prometheanfire I uninstalled nova/oslo-service/oslo-db system-wide
15:23:42 prometheanfire only available in the venv
15:23:48 sean-k-mooney kashyap: documenting it there is fine but the autritive source is glance
15:23:49 prometheanfire like I said, broken :P
15:24:13 stephenfin prometheanfire: Very :)
15:24:24 kashyap sean-k-mooney: Yep, noted. (And it seems to be valid for both, IIUC)
15:24:44 stephenfin prometheanfire: Yeah, I'm not sure how much I can help with that. There's something funky going on with venvs that I don't understand. I don't think it's anything to do with the patch itself
15:27:16 prometheanfire stephenfin: ya, at this point justmergeit
15:27:34 sean-k-mooney kashyap: doing a code search i only see code for using it from the conf or the image not the flavor
15:27:51 sean-k-mooney kashyap: http://codesearch.openstack.org/?q=machine_type&i=nope&files=&repos=nova
15:28:12 kashyap sean-k-mooney: Yeah, I've just done a test, indeed it's so
15:28:24 kashyap stephenfin: Hey, when you get a moment, yesterday you said this worked for you:
15:28:42 kashyap $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35
15:28:42 kashyap $ openstack flavor create test.q35
15:28:45 kashyap $ openstack server create --flavor test.q35 --image test \
15:28:48 kashyap --nic net-id=$NIC_UUID test-q35
15:29:00 kashyap stephenfin: Did the second command really take effect? It shouldn't have worked.
15:29:06 stephenfin kashyap: No, it didn't. I misread your comments
15:29:09 kashyap I guess you had 'q35' via other means, like config
15:29:10 stephenfin Hence the second set of pastes
15:29:10 kashyap Ah-ha!
15:29:13 sean-k-mooney kashyap: i think stephenfin had to use the config
15:31:08 kashyap Yep, it's all clear now.
15:58:47 openstackgerrit Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785
16:14:40 openstackgerrit Matt Riedemann proposed openstack/nova master: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599670
16:14:52 mriedem efried: bauzas: jaypipes: ^ follow up for config option wording
16:15:28 bauzas mriedem: thanks, and bingo
16:17:05 efried mriedem: +A, nice one guv
16:19:28 prometheanfire huh, nova destroyed an instance when I was messing with placement stuff
16:21:54 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/599672
16:21:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599673
16:22:20 jaypipes mriedem: I'd rather have someone like mgagne look at that patch and give advice.
16:22:26 jaypipes since we're not operators...
16:22:55 mriedem sure, hence the big todo comment in the bottom change
16:24:20 sean-k-mooney prometheanfire did you delete its allocation or something?
16:25:05 sean-k-mooney prometheanfire: i did not think we could kill nova instance by messing with placement so that sounds... unintended
16:25:23 prometheanfire sean-k-mooney: no, it looks like libvirt forgot it exists
16:25:39 prometheanfire or something
16:25:54 sean-k-mooney is the instace still listed in openstack. e.g. opestack server list
16:25:55 prometheanfire 2018-09-04 16:11:44.844 4079 INFO nova.compute.manager [req-ab55e9f6-b2a8-48ac-b50b-fde5b7af0892 - - - - -] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Deleting instance as it has been evacuated from this host
16:26:04 prometheanfire no clue why it was evacualted :|
16:26:19 prometheanfire that's the first log line on start of nova-compute
16:26:21 sean-k-mooney oh ok am ill go with ghosts
16:26:27 sean-k-mooney or your other admins
16:26:35 prometheanfire I'm the only admin :P
16:26:45 sean-k-mooney then ill stick with my first answer
16:26:57 prometheanfire what triggers a evacuation?
16:27:19 sean-k-mooney you dont have watcher deployed or one of the other ha serices do you?
16:27:27 prometheanfire nope
16:27:35 sean-k-mooney prometheanfire: as far as i knew evacuate was an admin only api call
16:27:56 sean-k-mooney so without manually invoking it i did not think we had a way to auto evacuate
16:28:25 prometheanfire ghosts then
16:28:38 sean-k-mooney mriedem: jaypipes dansmith any idea what could cause an evacuation to happen without an admin doing it?
16:29:04 mgagne jaypipes, mriedem: This change is already Workflow+1, wording is fine for me anyway. (re https://review.openstack.org/#/c/599670/)
16:30:06 prometheanfire sean-k-mooney: https://gist.githubusercontent.com/prometheanfire/76fc4693feed8b99118feaebbadfaea4/raw/c2a1b60e7181d7b864af35730e9f7eadfa365fd1/gistfile1.txt
16:30:32 prometheanfire sean-k-mooney: it looks like it tries to see if it was evacuated, gets the timeout/traceback and sees that as ok?
16:31:06 prometheanfire sean-k-mooney: github gist links do suck
16:31:44 prometheanfire 2018-09-04 16:34:15.499 4079 WARNING nova.compute.manager [req-2f9b1170-6748-4c94-af5f-00e8fc70d0e9 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
16:31:48 prometheanfire lol
16:31:49 sean-k-mooney prometheanfire: had you previously evacuated instaces from that host? if so when the nova compute agent comes back up it will clean up any instaces that were not deleted
16:31:56 prometheanfire is there a way to get nova to recreate the libvirt domain?
16:32:01 prometheanfire sean-k-mooney: nope
16:32:10 prometheanfire not that I can remember at all
16:32:39 jaypipes mgagne: yes, I recognize the change is already +W. I just wanted a real operator to double-check the wording. :)
16:32:52 sean-k-mooney prometheanfire: well it depends if you do opestack server show 0e9aa374-3627-48ac-a410-4abd65564a80 i assume it is gone?
16:33:06 mgagne jaypipes: +1
16:33:20 sean-k-mooney prometheanfire: if not then it should be running somewhere else in your cloud
16:34:20 prometheanfire sean-k-mooney: the server show still works, when I do a reboot I get this
16:34:23 prometheanfire 2018-09-04 16:23:57.779 4079 ERROR nova.compute.manager [req-e7f9f2a5-7cbc-4776-a085-88245450abac bcebdc7b8dfd4d43b036d1b73df6d377 5488a33661454bd792ff8c62d31d07a0 - default default] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Cannot reboot instance: Instance 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.: nova.exception.InstanceNotFound: Instance
16:34:28 prometheanfire 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.
16:35:18 sean-k-mooney prometheanfire: am from the admin view can you check the host its running on and see if libvirt see it?
16:36:37 prometheanfire virsh list --all doesn't show it
16:36:45 prometheanfire this had to be my dns server too
16:37:25 sean-k-mooney prometheanfire: im guessing its partially deleted. you could try a force reset of the vm status followed by a hard reboot
16:38:07 prometheanfire sean-k-mooney: where is the libvirt.xml stuff stored now?
16:38:21 prometheanfire I could recreate the domain and it'd probably work
16:39:30 prometheanfire /etc/libvirt/qemu/ it looks like
16:40:13 sean-k-mooney yes. if you know the instace name the xml might still be there if not the qemu args will be in /var/log/libvirt/qeum/instacne...

Earlier   Later