Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-04
16:14:52 mriedem efried: bauzas: jaypipes: ^ follow up for config option wording
16:15:28 bauzas mriedem: thanks, and bingo
16:17:05 efried mriedem: +A, nice one guv
16:19:28 prometheanfire huh, nova destroyed an instance when I was messing with placement stuff
16:21:54 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/599672
16:21:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599673
16:22:20 jaypipes mriedem: I'd rather have someone like mgagne look at that patch and give advice.
16:22:26 jaypipes since we're not operators...
16:22:55 mriedem sure, hence the big todo comment in the bottom change
16:24:20 sean-k-mooney prometheanfire did you delete its allocation or something?
16:25:05 sean-k-mooney prometheanfire: i did not think we could kill nova instance by messing with placement so that sounds... unintended
16:25:23 prometheanfire sean-k-mooney: no, it looks like libvirt forgot it exists
16:25:39 prometheanfire or something
16:25:54 sean-k-mooney is the instace still listed in openstack. e.g. opestack server list
16:25:55 prometheanfire 2018-09-04 16:11:44.844 4079 INFO nova.compute.manager [req-ab55e9f6-b2a8-48ac-b50b-fde5b7af0892 - - - - -] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Deleting instance as it has been evacuated from this host
16:26:04 prometheanfire no clue why it was evacualted :|
16:26:19 prometheanfire that's the first log line on start of nova-compute
16:26:21 sean-k-mooney oh ok am ill go with ghosts
16:26:27 sean-k-mooney or your other admins
16:26:35 prometheanfire I'm the only admin :P
16:26:45 sean-k-mooney then ill stick with my first answer
16:26:57 prometheanfire what triggers a evacuation?
16:27:19 sean-k-mooney you dont have watcher deployed or one of the other ha serices do you?
16:27:27 prometheanfire nope
16:27:35 sean-k-mooney prometheanfire: as far as i knew evacuate was an admin only api call
16:27:56 sean-k-mooney so without manually invoking it i did not think we had a way to auto evacuate
16:28:25 prometheanfire ghosts then
16:28:38 sean-k-mooney mriedem: jaypipes dansmith any idea what could cause an evacuation to happen without an admin doing it?
16:29:04 mgagne jaypipes, mriedem: This change is already Workflow+1, wording is fine for me anyway. (re https://review.openstack.org/#/c/599670/)
16:30:06 prometheanfire sean-k-mooney: https://gist.githubusercontent.com/prometheanfire/76fc4693feed8b99118feaebbadfaea4/raw/c2a1b60e7181d7b864af35730e9f7eadfa365fd1/gistfile1.txt
16:30:32 prometheanfire sean-k-mooney: it looks like it tries to see if it was evacuated, gets the timeout/traceback and sees that as ok?
16:31:06 prometheanfire sean-k-mooney: github gist links do suck
16:31:44 prometheanfire 2018-09-04 16:34:15.499 4079 WARNING nova.compute.manager [req-2f9b1170-6748-4c94-af5f-00e8fc70d0e9 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor.
16:31:48 prometheanfire lol
16:31:49 sean-k-mooney prometheanfire: had you previously evacuated instaces from that host? if so when the nova compute agent comes back up it will clean up any instaces that were not deleted
16:31:56 prometheanfire is there a way to get nova to recreate the libvirt domain?
16:32:01 prometheanfire sean-k-mooney: nope
16:32:10 prometheanfire not that I can remember at all
16:32:39 jaypipes mgagne: yes, I recognize the change is already +W. I just wanted a real operator to double-check the wording. :)
16:32:52 sean-k-mooney prometheanfire: well it depends if you do opestack server show 0e9aa374-3627-48ac-a410-4abd65564a80 i assume it is gone?
16:33:06 mgagne jaypipes: +1
16:33:20 sean-k-mooney prometheanfire: if not then it should be running somewhere else in your cloud
16:34:20 prometheanfire sean-k-mooney: the server show still works, when I do a reboot I get this
16:34:23 prometheanfire 2018-09-04 16:23:57.779 4079 ERROR nova.compute.manager [req-e7f9f2a5-7cbc-4776-a085-88245450abac bcebdc7b8dfd4d43b036d1b73df6d377 5488a33661454bd792ff8c62d31d07a0 - default default] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Cannot reboot instance: Instance 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.: nova.exception.InstanceNotFound: Instance
16:34:28 prometheanfire 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.
16:35:18 sean-k-mooney prometheanfire: am from the admin view can you check the host its running on and see if libvirt see it?
16:36:37 prometheanfire virsh list --all doesn't show it
16:36:45 prometheanfire this had to be my dns server too
16:37:25 sean-k-mooney prometheanfire: im guessing its partially deleted. you could try a force reset of the vm status followed by a hard reboot
16:38:07 prometheanfire sean-k-mooney: where is the libvirt.xml stuff stored now?
16:38:21 prometheanfire I could recreate the domain and it'd probably work
16:39:30 prometheanfire /etc/libvirt/qemu/ it looks like
16:40:13 sean-k-mooney yes. if you know the instace name the xml might still be there if not the qemu args will be in /var/log/libvirt/qeum/instacne...
16:44:06 prometheanfire ya, that helps some
16:44:27 prometheanfire the xml isn't there, but I can register a domain with it and it should be picked up then, hopefully...
16:44:45 kashyap stephenfin: Hey, if you still have that env, can I ask to do one last test, please?
16:44:58 sean-k-mooney prometheanfire: nova will rechreate the domain for you if you can start the vm.
16:45:06 kashyap stephenfin: It is the following:
16:45:31 kashyap stephenfin: Boot a guest w/ Q35, but now with _8_ PCIe root ports, using 'num_pcie_ports=8' in nova.conf
16:45:31 sean-k-mooney prometheanfire: the easies thing to do would be to rest the state to active. issue a shudown the reset to active agin if need and start the instace
16:45:40 prometheanfire sean-k-mooney: in that case reset state and start
16:45:43 prometheanfire ya
16:49:51 prometheanfire and we're back, odd that it happened, but ok
16:50:16 sean-k-mooney prometheanfire: was your instance on ceph or cinder storage?
16:50:43 prometheanfire no
16:50:47 prometheanfire just a basic instance
16:51:38 sean-k-mooney :( in that case the evacuate deleted your data as it did a force rebuild to a different host
16:52:12 prometheanfire sean-k-mooney: ssh was happy, so it didn't
16:52:19 prometheanfire no clue what happened, but I'm fine
16:52:30 prometheanfire even if it did, it's all in puppet for this node
16:53:57 prometheanfire my compute nodes still aren't reporting to placement though, not sure they ever did that right
16:53:58 sean-k-mooney prometheanfire: well ssh would be fixed via cloud init
16:54:30 sean-k-mooney if you data is intack however thats a good thing :)
16:54:47 prometheanfire I meant the host key is the same
16:56:43 prometheanfire sean-k-mooney: btw, did you see my comment about the database connection string in the apidb not being urlencoded?
16:56:46 openstackgerrit Merged openstack/nova master: Combine error handling blocks in _do_build_and_run_instance https://review.openstack.org/545960
16:57:09 prometheanfire too far in backlog now
16:58:21 prometheanfire update cell_mappings set database_connection='URLENCODED_CONNECTION_STRING' where uuid='0000'
17:01:05 SamYaple what actions, if any, can a user do that will trigger an 'instance.update' notification? having troble figuring it out from the code
17:03:24 prometheanfire There are no compute resource providers in the Placement service but there are 2 compute nodes in the deployment. This means no compute nodes are reporting into the Placement service and need to be upgraded and/or fixed.
17:03:40 prometheanfire but the compute nodes have the placement info in their config, not sure what else to do there
17:05:27 mriedem SamYaple: do you have notify_on_state_change set?
17:05:34 SamYaple mriedem: yes
17:05:43 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notify_on_state_change
17:05:44 mriedem to what?
17:05:57 SamYaple prometheanfire: just checked scrollback, i just dealt with that issue! on a few nodes. It was caused by reprovisioning my compute nodes with teh same names but migrating the instances off with evacuate. when it came back the database was messed up
17:06:46 mriedem prometheanfire: are you on pike or master/
17:06:47 mriedem ?
17:06:50 sean-k-mooney prometheanfire: jaypipes or cdent might be able to help with that. im guessiog your missing the placement client and or config section in the nova config but honestly that is jsut a guess
17:06:50 prometheanfire SamYaple: oh?
17:06:55 prometheanfire mriedem: rocky
17:07:42 SamYaple mriedem: so it *only* sends updates with state changes? if i read that correctly? (sorry, chasing down a bug in an inherited notifications reader with bad comments, so im trying to be explicit)
17:08:03 SamYaple prometheanfire: yes, it had to do with the old instances that had been in an ERROR state before the node went down
17:08:05 prometheanfire sean-k-mooney: there's a separate placement client? as in pypi type thing?
17:08:05 mriedem prometheanfire: i relatively recently talked with dansmith about trying to parse/encode/decode the db connection string in the cell_mappings table and i think the consensus was if you have special stuff in the url, you need to encode it beforehand
17:08:12 sean-k-mooney SamYaple: that kind of makes sense. since you used the same host name the compute agent will get teh compute servie record for the previous install
17:08:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556
17:08:14 SamYaple prometheanfire: ill try to pull my sql queries to fix it
17:08:30 prometheanfire mriedem: ah, somehow I got it in the db
17:09:51 sean-k-mooney prometheanfire: there is a placement osc plugin but when i said placement client i was referint to the devstack placement clinet service but honestly dont know what that does

Earlier   Later