| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 17:39:53 | sean-k-mooney | stephenfin: without a change to devstack nova would be relying on other pojects to install pymsql or psycopg2 for postgress | |
| 17:43:14 | efried | mriedem: any advice on next steps for copying mysql logs? https://review.opendev.org/#/c/678051/ | |
| 17:45:54 | efried | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1] | |
| 17:45:54 | efried | rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] | |
| 17:45:54 | efried | sudo: no tty present and no askpass program specified | |
| 17:45:54 | efried | Warning: Permanently added '166.78.116.9' (ECDSA) to the list of known hosts. | |
| 17:45:54 | efried | mriedem: | |
| 17:47:12 | sean-k-mooney | i dont think you want to copy them to the executer liek that i think you want to drop them into the working directory on the vm and have the normal log copy task copy them at the end | |
| 17:48:14 | sean-k-mooney | i may be wrong about that but i think that is what i have seen done before | |
| 17:49:53 | artom | sean-k-mooney, do you know if we would break anything if we started doing claims for all live migrations? I'm thinking of PCI stuff... | |
| 17:50:13 | artom | Presumably claims for PCI aren't broken in and of themselves, since they're used to boot instances and for cold migrations | |
| 17:50:31 | sean-k-mooney | we do claims for pci deivces for migration already | |
| 17:50:40 | artom | *live*? | |
| 17:50:43 | sean-k-mooney | yes | |
| 17:51:07 | sean-k-mooney | well for neutron sriov ports | |
| 17:51:17 | sean-k-mooney | we dont support live migration with other pci devices | |
| 17:51:30 | sean-k-mooney | but for sriov migration we added cliams | |
| 17:51:45 | sean-k-mooney | *sriov live migration | |
| 17:51:51 | artom | Are we talking about the same claims? | |
| 17:51:59 | sean-k-mooney | in the resouce tracker | |
| 17:52:03 | artom | Yeah | |
| 17:52:10 | sean-k-mooney | we do it as part of pre live migration at dest | |
| 17:52:11 | artom | Could you point me to the review/code? | |
| 17:52:22 | sean-k-mooney | lookign for it now | |
| 17:53:30 | sean-k-mooney | https://review.opendev.org/#/c/616120/ | |
| 17:53:41 | sean-k-mooney | and i think we use them in https://review.opendev.org/#/c/620115/28 | |
| 17:54:32 | sean-k-mooney | the pci manager works differently then the numa toplogy in the RT | |
| 17:54:59 | sean-k-mooney | but ehere is the claim in the compute manager | |
| 17:55:01 | sean-k-mooney | https://review.opendev.org/#/c/620115/28/nova/compute/manager.py | |
| 17:55:45 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 17:55:45 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 17:56:43 | artom | sean-k-mooney, it's not actually creating a MoveClaim object though, is it? | |
| 17:57:33 | artom | sean-k-mooney, I'm thinking of a scenario where we live-migration a NUMA instance with neutron SRIOV ports | |
| 17:57:50 | sean-k-mooney | no its not as far as i rememeber | |
| 17:57:53 | artom | Will the MoveClaim for the NUMA part conflict with the PCI claim for the SRIOV part | |
| 17:58:23 | sean-k-mooney | no we claim them in the rt and store the pci addresses in the live migration data object | |
| 17:58:31 | sean-k-mooney | i dont think we are using move claims at all | |
| 17:59:46 | sean-k-mooney | we might use move claims for cold migration i have not check that in a long time | |
| 17:59:52 | mriedem | efried: i think it's failing because the dest doesn't exist when your task runs | |
| 18:00:03 | mriedem | that's done later here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/5ba0f501-9e58-4f71-859a-ef084a19e0be/#line-11 | |
| 18:00:33 | artom | sean-k-mooney, if you have the hardware to test those 2 together it would put many minds at east, I think | |
| 18:00:33 | efried | mriedem: infra said it was because sudo is disabled for tox by default. Latest respin (above) reparents to openstack-tox-with-sudo. | |
| 18:00:58 | sean-k-mooney | i plan to do that yes | |
| 18:01:20 | sean-k-mooney | ill do it monday or tuesday if you like | |
| 18:01:34 | efried | mriedem: but maybe that too. Though I would have thought they would set up log dirs before invoking posts. | |
| 18:02:50 | sean-k-mooney | i have 1 of my sriov systems setup with 18.04 so i need to install 18.04 on the second one an then install centos on a third host to test something else | |
| 18:03:19 | sean-k-mooney | ill kick of those now and i should be able to test it monday if you bug me or tuesday if you dont | |
| 18:05:39 | artom | sean-k-mooney, appreciated :) | |
| 18:06:30 | mriedem | efried: maybe it is https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/c8a99499-4049-4878-b0a9-13a9292437f4/#line-1 | |
| 18:06:48 | mriedem | efried: so maybe your dest needs to be '{{ zuul_log_path }}' | |
| 18:06:53 | mriedem | er + /mysql/ | |
| 18:07:24 | mriedem | or {{ zuul_output_dir }}/logs/mysql | |
| 18:07:28 | mriedem | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/f31abc72-92ae-4e7d-b37c-8c124f7f32bd/#line-1 | |
| 18:09:55 | efried | mriedem: okay, can try something along those lines if this one borks again. | |
| 18:10:25 | mriedem | ok | |
| 18:10:31 | mriedem | you know about the ara report right? | |
| 18:10:39 | mriedem | to see what runs before and after you with what values | |
| 18:13:52 | sean-k-mooney | artom: once i have it up an running i can give you remote acess to poke around if you want. i have a static ip so i can easily port forwad you to i the hosts. ubuntu is install now so i just need to add a couple of kernel arges to make sriov work then stack with your code | |
| 18:14:07 | artom | sean-k-mooney, ack, that works | |
| 18:15:20 | artom | sean-k-mooney, I'm thinking of this bit in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L283-L295 | |
| 18:15:35 | artom | For NUMA LM, we call that during check_live_migration_at_destination | |
| 18:16:47 | efried | mriedem: no, I knew it was there, but never really understood what it meant. | |
| 18:18:55 | mriedem | btw, speaking of that nova.image.download module https://logs.opendev.org/84/676684/1/check/neutron-grenade-multinode/b0a76ac/logs/screen-n-api.txt#_Aug_15_13_25_04_744293 | |
| 18:18:59 | mriedem | i'm not sure why that's happening | |
| 18:19:05 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L294-L295 might be a proablem | |
| 18:19:31 | sean-k-mooney | since we have already claimed them on the dest it could claim them twice | |
| 18:19:50 | mriedem | oh gdi of course https://review.opendev.org/#/c/676684/ | |
| 18:19:51 | mriedem | hehh | |
| 18:19:56 | sean-k-mooney | i think currenlty we only use that for cold migration right | |
| 18:21:30 | artom | sean-k-mooney, *currently* | |
| 18:21:45 | artom | But NUMA LM is based on doing a MoveClaim | |
| 18:22:09 | sean-k-mooney | right so we would have to modify this or it wont work with sriov live migration | |
| 18:22:18 | sean-k-mooney | or rather might not | |
| 18:23:18 | sean-k-mooney | will you be passing in somthing to indicate its a live migration | |
| 18:23:20 | artom | Or just ignore pci_requests entirely if it's a live migration? | |
| 18:23:31 | sean-k-mooney | that is what i was going to suggest | |
| 18:23:33 | artom | Yeah, the migration will hava a type set | |
| 18:23:34 | openstackgerrit | Andreas Jaeger proposed openstack/nova master: PDF documentation build https://review.opendev.org/676730 | |
| 18:23:54 | sean-k-mooney | its nice that it is claiming alias based pci devices | |
| 18:24:02 | sean-k-mooney | we dont currently support that in the sriov migration | |
| 18:24:13 | sean-k-mooney | but that also means we woudl not update the xml | |
| 18:24:43 | artom | Yeah, we can't claim resource we have no intention of using | |
| 18:24:44 | sean-k-mooney | so what we need to to is 1 check that there is not alias based pci device and two not call the cpi claim bit if its a live migration | |
| 18:25:06 | artom | Why is 2 alone not enough? | |
| 18:25:30 | sean-k-mooney | it is but the other is a safty mesure | |
| 18:25:48 | sean-k-mooney | e.g. we shoudl not attemt to migrate if you have alias based pci passthough | |
| 18:25:54 | sean-k-mooney | its not supported at all | |
| 18:26:08 | sean-k-mooney | we might already check that somewhere | |
| 18:26:16 | sean-k-mooney | if we dont we should in the conductor | |
| 18:26:35 | artom | Seems like that should be separate from NUMA LM tho | |
| 18:26:44 | sean-k-mooney | ya it is. | |
| 18:27:10 | artom | NUMA LM just cares about not stepping on SRIOV LM's toes | |
| 18:27:15 | sean-k-mooney | so for numa the if(migration.type!=live) shoudl be enough | |
| 18:28:12 | artom | We have functional tests for SRIOV LM? | |
| 18:28:20 | artom | I wonder if we could try both in a func test | |
| 18:28:41 | sean-k-mooney | we dont | |
| 18:29:06 | artom | Func tests for neutron SRIOV at all? | |
| 18:29:27 | sean-k-mooney | nope | |
| 18:29:38 | sean-k-mooney | we have for flavor based pci passthough | |
| 18:29:43 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_pci_sriov_servers.py | |