Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-04
16:32:10 prometheanfire not that I can remember at all
16:32:39 jaypipes mgagne: yes, I recognize the change is already +W. I just wanted a real operator to double-check the wording. :)
16:32:52 sean-k-mooney prometheanfire: well it depends if you do opestack server show 0e9aa374-3627-48ac-a410-4abd65564a80 i assume it is gone?
16:33:06 mgagne jaypipes: +1
16:33:20 sean-k-mooney prometheanfire: if not then it should be running somewhere else in your cloud
16:34:20 prometheanfire sean-k-mooney: the server show still works, when I do a reboot I get this
16:34:23 prometheanfire 2018-09-04 16:23:57.779 4079 ERROR nova.compute.manager [req-e7f9f2a5-7cbc-4776-a085-88245450abac bcebdc7b8dfd4d43b036d1b73df6d377 5488a33661454bd792ff8c62d31d07a0 - default default] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Cannot reboot instance: Instance 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.: nova.exception.InstanceNotFound: Instance
16:34:28 prometheanfire 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.
16:35:18 sean-k-mooney prometheanfire: am from the admin view can you check the host its running on and see if libvirt see it?
16:36:37 prometheanfire virsh list --all doesn't show it
16:36:45 prometheanfire this had to be my dns server too
16:37:25 sean-k-mooney prometheanfire: im guessing its partially deleted. you could try a force reset of the vm status followed by a hard reboot
16:38:07 prometheanfire sean-k-mooney: where is the libvirt.xml stuff stored now?
16:38:21 prometheanfire I could recreate the domain and it'd probably work
16:39:30 prometheanfire /etc/libvirt/qemu/ it looks like
16:40:13 sean-k-mooney yes. if you know the instace name the xml might still be there if not the qemu args will be in /var/log/libvirt/qeum/instacne...
16:44:06 prometheanfire ya, that helps some
16:44:27 prometheanfire the xml isn't there, but I can register a domain with it and it should be picked up then, hopefully...
16:44:45 kashyap stephenfin: Hey, if you still have that env, can I ask to do one last test, please?
16:44:58 sean-k-mooney prometheanfire: nova will rechreate the domain for you if you can start the vm.
16:45:06 kashyap stephenfin: It is the following:
16:45:31 kashyap stephenfin: Boot a guest w/ Q35, but now with _8_ PCIe root ports, using 'num_pcie_ports=8' in nova.conf
16:45:31 sean-k-mooney prometheanfire: the easies thing to do would be to rest the state to active. issue a shudown the reset to active agin if need and start the instace
16:45:40 prometheanfire sean-k-mooney: in that case reset state and start
16:45:43 prometheanfire ya
16:49:51 prometheanfire and we're back, odd that it happened, but ok
16:50:16 sean-k-mooney prometheanfire: was your instance on ceph or cinder storage?
16:50:43 prometheanfire no
16:50:47 prometheanfire just a basic instance
16:51:38 sean-k-mooney :( in that case the evacuate deleted your data as it did a force rebuild to a different host
16:52:12 prometheanfire sean-k-mooney: ssh was happy, so it didn't
16:52:19 prometheanfire no clue what happened, but I'm fine
16:52:30 prometheanfire even if it did, it's all in puppet for this node
16:53:57 prometheanfire my compute nodes still aren't reporting to placement though, not sure they ever did that right
16:53:58 sean-k-mooney prometheanfire: well ssh would be fixed via cloud init
16:54:30 sean-k-mooney if you data is intack however thats a good thing :)
16:54:47 prometheanfire I meant the host key is the same
16:56:43 prometheanfire sean-k-mooney: btw, did you see my comment about the database connection string in the apidb not being urlencoded?
16:56:46 openstackgerrit Merged openstack/nova master: Combine error handling blocks in _do_build_and_run_instance https://review.openstack.org/545960
16:57:09 prometheanfire too far in backlog now
16:58:21 prometheanfire update cell_mappings set database_connection='URLENCODED_CONNECTION_STRING' where uuid='0000'
17:01:05 SamYaple what actions, if any, can a user do that will trigger an 'instance.update' notification? having troble figuring it out from the code
17:03:24 prometheanfire There are no compute resource providers in the Placement service but there are 2 compute nodes in the deployment. This means no compute nodes are reporting into the Placement service and need to be upgraded and/or fixed.
17:03:40 prometheanfire but the compute nodes have the placement info in their config, not sure what else to do there
17:05:27 mriedem SamYaple: do you have notify_on_state_change set?
17:05:34 SamYaple mriedem: yes
17:05:43 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notify_on_state_change
17:05:44 mriedem to what?
17:05:57 SamYaple prometheanfire: just checked scrollback, i just dealt with that issue! on a few nodes. It was caused by reprovisioning my compute nodes with teh same names but migrating the instances off with evacuate. when it came back the database was messed up
17:06:46 mriedem prometheanfire: are you on pike or master/
17:06:47 mriedem ?
17:06:50 sean-k-mooney prometheanfire: jaypipes or cdent might be able to help with that. im guessiog your missing the placement client and or config section in the nova config but honestly that is jsut a guess
17:06:50 prometheanfire SamYaple: oh?
17:06:55 prometheanfire mriedem: rocky
17:07:42 SamYaple mriedem: so it *only* sends updates with state changes? if i read that correctly? (sorry, chasing down a bug in an inherited notifications reader with bad comments, so im trying to be explicit)
17:08:03 SamYaple prometheanfire: yes, it had to do with the old instances that had been in an ERROR state before the node went down
17:08:05 prometheanfire sean-k-mooney: there's a separate placement client? as in pypi type thing?
17:08:05 mriedem prometheanfire: i relatively recently talked with dansmith about trying to parse/encode/decode the db connection string in the cell_mappings table and i think the consensus was if you have special stuff in the url, you need to encode it beforehand
17:08:12 sean-k-mooney SamYaple: that kind of makes sense. since you used the same host name the compute agent will get teh compute servie record for the previous install
17:08:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556
17:08:14 SamYaple prometheanfire: ill try to pull my sql queries to fix it
17:08:30 prometheanfire mriedem: ah, somehow I got it in the db
17:09:51 sean-k-mooney prometheanfire: there is a placement osc plugin but when i said placement client i was referint to the devstack placement clinet service but honestly dont know what that does
17:09:58 mriedem SamYaple: it looks like a hack way to trigger the instance.update notififcation is to change metadata on the instance
17:10:00 SamYaple sean-k-mooney: would a possible workaround for my use case possibly be solved simply by deleting the service record between reprovisioning?
17:10:32 SamYaple mriedem: ah! perfect! ok. so thats the hack this code is trying to tap into. that aligns on my end. thank you
17:10:34 prometheanfire sean-k-mooney: ok, thought the nova-compute service handled registering itself
17:10:49 mriedem prometheanfire: it does
17:10:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556
17:10:55 mriedem SamYaple: which release are you on?
17:11:17 SamYaple mriedem: ocata/pike/queens across a few environments
17:11:27 sean-k-mooney SamYaple:ya perhapos never tried it. with all the forign key on the table however it might be more hassel then its worth
17:11:38 mriedem ok, because we didn't until recently cleanup after ourselves wrt placement when deleting a nova-compute service record
17:11:49 mriedem thta's fixed now, but you'd have to make sure you have it, otherwise you have to cleanup placement entries yourself
17:12:02 sean-k-mooney prometheanfire: if you have the right setting in your conf the nova-compute agent will take care of populating the placement api for you
17:12:20 SamYaple ah, yes. this process has only been done on the older ocata clusters. We should be queens everywhere by end of year though.
17:12:28 prometheanfire sean-k-mooney: nova-status upgrade check still shows the warning
17:12:31 SamYaple mriedem: my sql are me cleaning up the placement entries :)
17:12:43 mriedem fyi https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80
17:13:49 SamYaple mriedem: oh b-e-a-utiful. Super helpful. gonna do an internal backport on that for our ocata stuff. got a few hundred more nodes to go
17:13:53 sean-k-mooney prometheanfire: for devstack the plamcenet section looks like this http://paste.openstack.org/show/729431/
17:14:28 prometheanfire sean-k-mooney: yep, looks right (looks like mine)
17:15:22 sean-k-mooney prometheanfire: strange and openstack resource provider list is does not have the compute nodes?
17:15:46 prometheanfire sean-k-mooney: http://paste.openstack.org/show/729432/
17:16:04 mriedem SamYaple: well, be warned, it's a hefty series of backports
17:16:28 mriedem this is just pike https://review.openstack.org/#/q/topic:bug/1756179+(status:open+OR+status:merged)+branch:stable/pike
17:16:30 prometheanfire sean-k-mooney: openstack service provider list
17:16:32 prometheanfire that?
17:16:49 sean-k-mooney no "openstack resource provider list"
17:17:05 sean-k-mooney if you do not have that you are missing the placement osc plugin
17:17:16 prometheanfire ok
17:17:21 mriedem prometheanfire: are there errors in the nova-compute logs?
17:17:44 prometheanfire mriedem: which errors you looking for?
17:17:44 sean-k-mooney prometheanfire: https://github.com/openstack/osc-placement for future reference. it should be on pypi too
17:17:52 mriedem prometheanfire: umm, any errors?
17:18:14 mriedem presumably "failed to create resource provider in placement" or something like that
17:18:16 prometheanfire mriedem: none now, I rebuilt all my systems from the ground up to fix the odd python issue
17:18:23 SamYaple mriedem: sorry, i said backport. i mean hack something in based on that commit. i stay pretty close to upstream
17:18:40 prometheanfire mriedem: testing now via ' su - nova -c "/usr/lib/python-exec/python3.5/nova-compute --config-file /etc/nova/nova-compute.conf --config-file /etc/nova/nova.conf" '
17:19:03 mriedem SamYaple: ok, well, good luck. :) if that series of backports on pike also applies to ocata (i'm not sure if they all do), we could also backport them to ocata upstream

Earlier   Later