| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 15:27:16 | prometheanfire | stephenfin: ya, at this point justmergeit | |
| 15:27:34 | sean-k-mooney | kashyap: doing a code search i only see code for using it from the conf or the image not the flavor | |
| 15:27:51 | sean-k-mooney | kashyap: http://codesearch.openstack.org/?q=machine_type&i=nope&files=&repos=nova | |
| 15:28:12 | kashyap | sean-k-mooney: Yeah, I've just done a test, indeed it's so | |
| 15:28:24 | kashyap | stephenfin: Hey, when you get a moment, yesterday you said this worked for you: | |
| 15:28:42 | kashyap | $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35 | |
| 15:28:42 | kashyap | $ openstack flavor create test.q35 | |
| 15:28:45 | kashyap | $ openstack server create --flavor test.q35 --image test \ | |
| 15:28:48 | kashyap | --nic net-id=$NIC_UUID test-q35 | |
| 15:29:00 | kashyap | stephenfin: Did the second command really take effect? It shouldn't have worked. | |
| 15:29:06 | stephenfin | kashyap: No, it didn't. I misread your comments | |
| 15:29:09 | kashyap | I guess you had 'q35' via other means, like config | |
| 15:29:10 | stephenfin | Hence the second set of pastes | |
| 15:29:10 | kashyap | Ah-ha! | |
| 15:29:13 | sean-k-mooney | kashyap: i think stephenfin had to use the config | |
| 15:31:08 | kashyap | Yep, it's all clear now. | |
| 15:58:47 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | |
| 16:14:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599670 | |
| 16:14:52 | mriedem | efried: bauzas: jaypipes: ^ follow up for config option wording | |
| 16:15:28 | bauzas | mriedem: thanks, and bingo | |
| 16:17:05 | efried | mriedem: +A, nice one guv | |
| 16:19:28 | prometheanfire | huh, nova destroyed an instance when I was messing with placement stuff | |
| 16:21:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/599672 | |
| 16:21:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599673 | |
| 16:22:20 | jaypipes | mriedem: I'd rather have someone like mgagne look at that patch and give advice. | |
| 16:22:26 | jaypipes | since we're not operators... | |
| 16:22:55 | mriedem | sure, hence the big todo comment in the bottom change | |
| 16:24:20 | sean-k-mooney | prometheanfire did you delete its allocation or something? | |
| 16:25:05 | sean-k-mooney | prometheanfire: i did not think we could kill nova instance by messing with placement so that sounds... unintended | |
| 16:25:23 | prometheanfire | sean-k-mooney: no, it looks like libvirt forgot it exists | |
| 16:25:39 | prometheanfire | or something | |
| 16:25:54 | sean-k-mooney | is the instace still listed in openstack. e.g. opestack server list | |
| 16:25:55 | prometheanfire | 2018-09-04 16:11:44.844 4079 INFO nova.compute.manager [req-ab55e9f6-b2a8-48ac-b50b-fde5b7af0892 - - - - -] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Deleting instance as it has been evacuated from this host | |
| 16:26:04 | prometheanfire | no clue why it was evacualted :| | |
| 16:26:19 | prometheanfire | that's the first log line on start of nova-compute | |
| 16:26:21 | sean-k-mooney | oh ok am ill go with ghosts | |
| 16:26:27 | sean-k-mooney | or your other admins | |
| 16:26:35 | prometheanfire | I'm the only admin :P | |
| 16:26:45 | sean-k-mooney | then ill stick with my first answer | |
| 16:26:57 | prometheanfire | what triggers a evacuation? | |
| 16:27:19 | sean-k-mooney | you dont have watcher deployed or one of the other ha serices do you? | |
| 16:27:27 | prometheanfire | nope | |
| 16:27:35 | sean-k-mooney | prometheanfire: as far as i knew evacuate was an admin only api call | |
| 16:27:56 | sean-k-mooney | so without manually invoking it i did not think we had a way to auto evacuate | |
| 16:28:25 | prometheanfire | ghosts then | |
| 16:28:38 | sean-k-mooney | mriedem: jaypipes dansmith any idea what could cause an evacuation to happen without an admin doing it? | |
| 16:29:04 | mgagne | jaypipes, mriedem: This change is already Workflow+1, wording is fine for me anyway. (re https://review.openstack.org/#/c/599670/) | |
| 16:30:06 | prometheanfire | sean-k-mooney: https://gist.githubusercontent.com/prometheanfire/76fc4693feed8b99118feaebbadfaea4/raw/c2a1b60e7181d7b864af35730e9f7eadfa365fd1/gistfile1.txt | |
| 16:30:32 | prometheanfire | sean-k-mooney: it looks like it tries to see if it was evacuated, gets the timeout/traceback and sees that as ok? | |
| 16:31:06 | prometheanfire | sean-k-mooney: github gist links do suck | |
| 16:31:44 | prometheanfire | 2018-09-04 16:34:15.499 4079 WARNING nova.compute.manager [req-2f9b1170-6748-4c94-af5f-00e8fc70d0e9 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor. | |
| 16:31:48 | prometheanfire | lol | |
| 16:31:49 | sean-k-mooney | prometheanfire: had you previously evacuated instaces from that host? if so when the nova compute agent comes back up it will clean up any instaces that were not deleted | |
| 16:31:56 | prometheanfire | is there a way to get nova to recreate the libvirt domain? | |
| 16:32:01 | prometheanfire | sean-k-mooney: nope | |
| 16:32:10 | prometheanfire | not that I can remember at all | |
| 16:32:39 | jaypipes | mgagne: yes, I recognize the change is already +W. I just wanted a real operator to double-check the wording. :) | |
| 16:32:52 | sean-k-mooney | prometheanfire: well it depends if you do opestack server show 0e9aa374-3627-48ac-a410-4abd65564a80 i assume it is gone? | |
| 16:33:06 | mgagne | jaypipes: +1 | |
| 16:33:20 | sean-k-mooney | prometheanfire: if not then it should be running somewhere else in your cloud | |
| 16:34:20 | prometheanfire | sean-k-mooney: the server show still works, when I do a reboot I get this | |
| 16:34:23 | prometheanfire | 2018-09-04 16:23:57.779 4079 ERROR nova.compute.manager [req-e7f9f2a5-7cbc-4776-a085-88245450abac bcebdc7b8dfd4d43b036d1b73df6d377 5488a33661454bd792ff8c62d31d07a0 - default default] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Cannot reboot instance: Instance 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.: nova.exception.InstanceNotFound: Instance | |
| 16:34:28 | prometheanfire | 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found. | |
| 16:35:18 | sean-k-mooney | prometheanfire: am from the admin view can you check the host its running on and see if libvirt see it? | |
| 16:36:37 | prometheanfire | virsh list --all doesn't show it | |
| 16:36:45 | prometheanfire | this had to be my dns server too | |
| 16:37:25 | sean-k-mooney | prometheanfire: im guessing its partially deleted. you could try a force reset of the vm status followed by a hard reboot | |
| 16:38:07 | prometheanfire | sean-k-mooney: where is the libvirt.xml stuff stored now? | |
| 16:38:21 | prometheanfire | I could recreate the domain and it'd probably work | |
| 16:39:30 | prometheanfire | /etc/libvirt/qemu/ it looks like | |
| 16:40:13 | sean-k-mooney | yes. if you know the instace name the xml might still be there if not the qemu args will be in /var/log/libvirt/qeum/instacne... | |
| 16:44:06 | prometheanfire | ya, that helps some | |
| 16:44:27 | prometheanfire | the xml isn't there, but I can register a domain with it and it should be picked up then, hopefully... | |
| 16:44:45 | kashyap | stephenfin: Hey, if you still have that env, can I ask to do one last test, please? | |
| 16:44:58 | sean-k-mooney | prometheanfire: nova will rechreate the domain for you if you can start the vm. | |
| 16:45:06 | kashyap | stephenfin: It is the following: | |
| 16:45:31 | sean-k-mooney | prometheanfire: the easies thing to do would be to rest the state to active. issue a shudown the reset to active agin if need and start the instace | |
| 16:45:31 | kashyap | stephenfin: Boot a guest w/ Q35, but now with _8_ PCIe root ports, using 'num_pcie_ports=8' in nova.conf | |
| 16:45:40 | prometheanfire | sean-k-mooney: in that case reset state and start | |
| 16:45:43 | prometheanfire | ya | |
| 16:49:51 | prometheanfire | and we're back, odd that it happened, but ok | |
| 16:50:16 | sean-k-mooney | prometheanfire: was your instance on ceph or cinder storage? | |
| 16:50:43 | prometheanfire | no | |
| 16:50:47 | prometheanfire | just a basic instance | |
| 16:51:38 | sean-k-mooney | :( in that case the evacuate deleted your data as it did a force rebuild to a different host | |
| 16:52:12 | prometheanfire | sean-k-mooney: ssh was happy, so it didn't | |
| 16:52:19 | prometheanfire | no clue what happened, but I'm fine | |
| 16:52:30 | prometheanfire | even if it did, it's all in puppet for this node | |
| 16:53:57 | prometheanfire | my compute nodes still aren't reporting to placement though, not sure they ever did that right | |
| 16:53:58 | sean-k-mooney | prometheanfire: well ssh would be fixed via cloud init | |
| 16:54:30 | sean-k-mooney | if you data is intack however thats a good thing :) | |
| 16:54:47 | prometheanfire | I meant the host key is the same | |
| 16:56:43 | prometheanfire | sean-k-mooney: btw, did you see my comment about the database connection string in the apidb not being urlencoded? | |
| 16:56:46 | openstackgerrit | Merged openstack/nova master: Combine error handling blocks in _do_build_and_run_instance https://review.openstack.org/545960 | |
| 16:57:09 | prometheanfire | too far in backlog now | |
| 16:58:21 | prometheanfire | update cell_mappings set database_connection='URLENCODED_CONNECTION_STRING' where uuid='0000' | |
| 17:01:05 | SamYaple | what actions, if any, can a user do that will trigger an 'instance.update' notification? having troble figuring it out from the code | |
| 17:03:24 | prometheanfire | There are no compute resource providers in the Placement service but there are 2 compute nodes in the deployment. This means no compute nodes are reporting into the Placement service and need to be upgraded and/or fixed. | |
| 17:03:40 | prometheanfire | but the compute nodes have the placement info in their config, not sure what else to do there | |
| 17:05:27 | mriedem | SamYaple: do you have notify_on_state_change set? | |